PS4-02: ASSESSING SURGERY RESIDENT AUTONOMY USING CUSUM: A NOVEL METHODOLOGY FOR VISUALIZING PROGRESSION TO OPERATIVE INDEPENDENCE
Ezra N Teitelbaum, MD, MEd1, Reiping Huang, PhD1, Shari L Meyerson, MD, MEd1, Mary C Schuller, MSEd1, Brian C George, MD, MEd2, Debra A DaRosa, PhD1, Jonathan P Fryer, MD, MHPE1, Yue-Yung Hu, MD, MPH3; 1Northwestern University, 2University of Michigan, 3Lurie Children's Hospital
Background: Concerns exist that some surgical residents are not capable of operating independently upon graduation. The comparative progression of individual residents towards operative independence is not well described. We adapted cumulative sum (CUSUM) methodology, used in manufacturing processes for ongoing analysis of accumulating data in real-time, to construct resident autonomy curves and assess variation between and within PGY-levels.
Methods: Faculty at an academic program rated operative autonomy of surgery residents using the Zwisch scale, which describes attending behaviors (show and tell, active help, passive help, supervision only). Prior to data collection, attendings were surveyed regarding the Zwisch level expected of each PGY-level for the 20 most frequent operations in their sub-specialty. Two-sided observed-expected CUSUM charts were then constructed to delineate each resident’s progression, with each “success” defined as meeting or exceeding the operative attending’s survey expectation and “failure” as performing below that expectation. CUSUM control limits were set using PGY-specific median and 75 percentile failure rates, to identify irregularly poor performers. Fisher’s exact tests were used to compare categorical outcomes.
Results: During the last 9 months of the 2012-13 academic year, 1,117 Zwisch ratings of 25 residents were collected and matched to faculty expectations (median 51/resident). Overall, residents met or exceeded attendings’ expectations in 46% of operations. PGY-5s had the lowest success rate in achieving expected autonomy (37% vs 51% for all others, p<0.0001). PGY-1s had the highest success rate (73% vs 45% for all others, p<0.0001). CUSUM charts (Figure) were generated to measure variation between residents within each PGY-class. There were significant differences between the highest and lowest performing members of each class, and one PGY-4 and one PGY-5 strayed outside the control limits for poor performance. Among PGY-5s, the top resident met expectations during 55% of operations, compared with 24% for the lowest (p<0.001).
Conclusions: Residents failed to meet attendings’ expectations for autonomy during more than half of operations. Interns most often met expectations, whereas chief residents had the lowest rate. Within each class there was substantial variation in progression. CUSUM holds significant promise as a way to benchmark residents and provide data about operative progress in real-time.
