The Ophthalmic Knowledge Assessment Program (OKAP) Examination and Global Evaluation of Resident Per
2008,Volume 1, Number 1 OKAP Examination - Graff 21 The Accreditation Council for Graduate Medical Educa-tion (ACGME) has mandated the implementation of six new competencies in the United States resident physi-cian training experience (patient care, medical knowl-edge, professionalism, communication and interpersonal skills, practice based learning, and systems based learn-ing) [2]. Medical knowledge is only one of the six ACGME competencies , but it has traditionally been the most emphasized and evaluated area of education. In the field of ophthalmology, the ABO WQE is the best studied post-training assessment of this compe-tency. For evaluation of medical knowledge during the training process, the OKAP examination is widely con-sidered to be a marker of fund of knowledge for resi-dents. The OKAP score is emerging as a high stakes encounter for residents in training and will likely be increasingly relied upon as a direct or indirect measure of the competency of medical knowledge. The goals of this study are to review the designed pur-pose and use of the OKAP examination and to compare the OKAP exam scores with global faculty evaluations of resident knowledge and performance at our institu-tion. Our study was prompted by the anecdotal obser-vation at our institution that residents with the highest OKAP score may not be the best resident as judged by our global faculty evaluations. Methods OKAP examination scores for 15 ophthalmology resi-dent physicians over a 3 year period of time (2003-2006) were collected from a single academic ophthal-mology program at The University of Iowa Hospitals and Clinics. We only used the OKAP results from resi-dent physicians in their second or third year of resi-dency training. The OKAP percentile rank used in the study was the total percentile score given in comparison with other residents in the same year of training taking the examination. During the same 3 year period, data was collected on the numerical global evaluation score awarded by supervising faculty physicians for the same resident physicians (E*Value electronic evaluation system, Advanced Informatics, LLC). This faculty global evaluation was a standardized evaluation (using a traditional Likert scale) which has not been formally tested for validity; however, the faculty were very fa-miliar with the system having used a similar norm-referenced scoring rubric for many years. Numerical scores were awarded in several areas of resident per-formance, including medical knowledge, application of knowledge in a clinical setting, understanding of rele-vant research, and clinical decision making, among oth-ers. The cumulative faculty global evaluation scores for each of the three years were totaled and the mean per-centage calculated as a percentage of a theoretical per-fect score (100%). All grading faculty (except the pro-gram director and program chair) were masked as to the results of OKAP examination. Figure 1 shows the com-parative data for OKAP score and E*Value score us-ing the Pearson's correlation coefficient (or). This study did not require institutional review board (IRB) approval but was IRB reviewed. Individual subject OKAP scores and faculty evaluations were compared anonymously and are reported in this paper without per-sonal identifiers in order to protect the privacy of the individuals. Results The mean OKAP examination percentile rank for the 15 residents evaluated was 51%, with a median percentile rank of 52%. The mean masked faculty score of resi-dent overall performance was 71% with a median per-centile score of 69%. There was no correlation between OKAP examination results and masked faculty scoring of resident overall performance ( = 0.11, see Figure 1). Interestingly, the resident with the highest OKAP per-centile rank of 92% scored towards the median score on faculty assessment of resident performance, with a score of 72%. There were several prominent outliers noted with the resident who received the highest marks from the faculty for overall clinical performance over 3 years (83%) achieving only a modest 57% rank on the OKAP examination. Likewise one resident who received an average (53%) faculty evaluations was well below aver-age scores (31%) on the OKAP exam. Evaluation Average vs OKAP ( r = .11) 50 60 70 80 90 20 40 60 80 100 OKAP % E* Va lu e Sc o re P e rcen tag e Evaluation Average vs OKAP Scores. There was no correlation between OKAP examination results and masked faculty scoring of resident overall performance, = 0.11.