Kaohsiung Journal of Medical Sciences
Volume 24, Issue 12 , Pages 627-633, December 2008

Evaluation of the Effectiveness of Postgraduate General Medicine Training by Objective Structured Clinical Examination—Pilot Study and Reflection on the Experiences of Kaohsiung Medical University Hospital

  • Jer-Chia Tsai

      Affiliations

    • College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    • Department of Internal Medicine, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
    • ,
  • Keh-Min Liu

      Affiliations

    • College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    • ,
  • Kun-Tai Lee

      Affiliations

    • College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    • Department of Internal Medicine, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
    • ,
  • Jo-Chu Yen

      Affiliations

    • Department of Clinical Education and Training, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
    • ,
  • Jeng-Hsien Yen

      Affiliations

    • College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    • Department of Internal Medicine, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
    • ,
  • Ching-Kuan Liu

      Affiliations

    • College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    • Department of Neurology, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
    • ,
  • Chung-Sheng Lai

      Affiliations

    • College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan
    • Department of Surgery, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
    • Corresponding Author InformationAddress correspondence and reprint requests to: Dr Chung-Sheng Lai, College of Medicine, Kaohsiung Medical University, 100 Shih-Chuan 1st Road, Kaohsiung 807, Taiwan

Received 5 January 2009; accepted 22 January 2009.

Article Outline

Objective structured clinical examination (OSCE) is an effective assessment method to evaluate medical students' clinical competencies performance. Postgraduate year 1 (PGY1) residents have been initiated in a general medicine training program in Taiwan since 2003. However, little is known about the learning effectiveness of trainees from this program. This pilot study aimed to evaluate the clinical core competencies of PGY1 residents using OSCE, and to reflect on the strengths and weaknesses of this pilot assessment project. OSCE was conducted for five PGY1 examinees (4 men, 1 woman) with five stations covering core themes, including history taking, physical examination, clinical procedure of airway intubation, clinical reasoning, and communication skills for informing bad news. Itemized checklists and five-point Likert scale global ratings were used for evaluating performance. The results showed that the performance of our PGY1 residents on history taking was significantly better after about 2 months of postgraduate training on general internal medicine. Self-evaluation on performance by examinees revealed significantly lower global ratings on post-course OSCE (4.14 ± 0.80 vs. 3.68 ± 0.66; p < 0.02). Surveys from tutors and standardized patients (SPs) completed at pre- and post-course OSCEs showed consistently favorable responses on the purposes, content, process, and environment of this assessment (4.0 ± 0.17 vs. 4.0 ± 0.12, nonsignificant). However, a survey of the examinees completed at preand post-course OSCEs showed relatively unfavorable responses to the same aspects, and to tutors and SPs (4.1 ± 0.09 vs. 3.7 ± 0.18; p < 0.05). Qualitative information revealed that tutors and SPs remarked that PGY1 residents' medical knowledge performance was satisfactory but their clinical reasoning performance, communication skills (giving bad news) and self-confidence were unsatisfactory. In conclusion, this pilot study has demonstrated that OSCE is a rational and feasible assessment method for evaluating the effectiveness of our PGY general medicine training program. The quantitative data and qualitative information provide a foundation to improve the quality of the program design and evaluation in implementing postgraduate general medicine training.

Key Words:  assessment , clinical competence , objective structured clinical examination (OSCE) , postgraduate medical education

No full text is available. To read the body of this article, please view the PDF online.

 

Back to Article Outline

References 

  1. Epstein RM . Assessment in medical education . N Engl J Med . 2007;356:387–396
  2. Norcini JJ , McKinley DW . Assessment methods in medical education . Teach Teacher Educ . 2007;23:239–250
  3. van der Vleuten CPM . The assessment of professional competence: developments, research and practical implications . Adv Health Sci Educ . 1996;1:41–67
  4. Harden RM , Gleeson FA . Assessment of clinical competence using an objective structured clinical examination (OSCE) . Med Educ . 1979;13:41–54
  5. Harden RM , Stevenson M , Wilson Downie W , et al.   Assessment of clinical competence using objective structured examination . BMJ . 1975;1:447–451
  6. Boursicot K , Roberts T . How to set up an OSCE . Clin Teach . 2005;2:16–20
  7. Stillman PL , Swanson DB , Smee S . Assessing clinical skills of residents with standardized patients . Ann Intern Med . 1986;105:762–771
  8. van der Vleuten C , Swanson DB . Assessment of clinical skills with standardized patients: state of the art . Teach Learn Med . 1990;2:58–76
  9. van Zanten M , Boulet J , Norcini J , et al.   Using a standardised patient assessment to measure professional attributes . Med Educ . 2005;39:20–29
  10. Barman A . Critiques on the objective structured clinical examination . Ann Acad Med Singapore . 2005;34:478–482
  11. Newble D . Techniques for measuring clinical competence: objective structured clinical examinations . Med Educ . 2004;38:199–203
  12. Miller GE . The assessment of clinical skills/competence/performance . Acad Med . 1990;65(9 Suppl):S63–S67
  13. Huang YS , Liu M , Huang CH , et al.   Implementation of an OSCE at Kaohsiung Medical University . Kaohsiung J Med Sci . 2007;23:161–169
  14. ACGME Accreditation Council for Graduate Medical Education  . Outcome Project, 2008 . Available from www.acgme.org/outcome/assess/assHome.asp [Date accessed: December 1, 2008]
  15. Swing SR . The ACGME outcome project: retrospective and prospective . Med Teach . 2007;29:648–654
  16. van der Vleuten C , Schuwirth L . Assessing professional competence: from methods to programmes . Med Educ . 2005;39:309–317
  17. Rutala PJ , Witzke DB , Leko EO , et al.   Student fatigue as a variable affecting performance in an objective structured clinical examination . Acad Med . 1990;65(9 Suppl):S53–S54
  18. Petrusa ER , Blackwell TA , Ainsworth MA . Reliability and validity of an objective structured clinical examination for assessing the clinical performance of residents . Arch Intern Med . 1990;150:573–577
  19. Regehr G , MacRae H , Reznick RK , et al.   Comparing the psychometric properties of checklists and global ratings scales for assessing performance on an OSCE-format examination . Acad Med . 1998;73:993–997
  20. Davis DA , Mazmanian PE , Fordis M , et al.   Accuracy of physician self-assessment compared with observed measures of competence: a systematic review . JAMA . 2006;296:1094–1102
  21. Nuovo J , Bertakis KD , Azari R . Assessing resident's knowledge and communication skills using four different evaluation tools . Med Educ . 2006;40:630–636
  22. Schuwirth LWT , van der Vleuten CPM . Changing education, changing assessment, changing research? . Med Educ . 2004;38:805–812
  23. Newble DI , Swanson DB . Psychometric characteristics of the objective structured clinical examination . Med Educ . 1988;22:325–334
  24. Wass V , van der Vleuten C , Shatzer J , et al.   Assessment of clinical competence . Lancet . 2001;357:945–949

PII: S1607-551X(09)70027-0

doi:10.1016/S1607-551X(09)70027-0

Kaohsiung Journal of Medical Sciences
Volume 24, Issue 12 , Pages 627-633, December 2008

Article Tools