Poster3 - 08: SURGERY CLERKSHIP STUDENTS DEMONSTRATE LARGE VARIABILITY IN CAPABILITY OF MANAGING CLINICAL SCENARIOS - A QUANTITATIVE ANALYSIS OF 1,737 VIRTUAL PATIENT CASE ATTEMPTS
Kelsey Hobart, BA, MS, Huaquin Zhao, PhD, Richard Bell, MD, John Daly, MD; Lewis Katz School of Medicine at Temple University

Introduction: Virtual Patients (VPs) are interactive computer-based simulations of patient scenarios in which users perform diagnostic and management tasks and make decisions as in real life. We performed a retrospective quantitative analysis of de-identified performance data of third year surgery clerks on ten VP cases over a two-year period (AYs 2016 and 2017).  This study was approved by the Institutional IRB.

Methods: VP cases were released to students online for one week to be done at the time of the student’s choosing. A computer generated score and feedback was given at the end of each attempt. At the end of the week, the students as a group met with one of us (JD) to review the case. Case completion was encouraged but not required and did not count towards the student’s clerkship grade.

Results: Surgery clerks completed 1,737 case attempts. Of these, 1,333 were first attempts and the remainder were repetitions. Students completed an average of 6 cases. Mean score on the initial attempt ranged from a low of 10 (out of possible 100) on the hardest case to a high of 61 on the easiest. There was high variability in scores on all cases, greater on the harder cases.  We examined other potential sources of variability. Mean score was not affected by the rotation number (6 cycles per year). There was evidence of improvement based on number of cases previously taken up to 4 weeks but not beyond. Students who ultimately matched in surgery residencies did not perform better than those who chose non-surgical residencies. For 3 of the 10 cases, there was a significant positive correlation between time spent on the case (up to 30 minutes) and score. Finally, scores improved from a mean 30.4 on all cases to 63.9 when a case was re-taken.

Conclusion: There was high variability in student scores on clinical scenarios. The variation is primarily due to student competency and not to extrinsic variables introduced by the virtual patient assessment methodology. More difficult cases accentuate the differences among students.