C2B - 04: DO SUMMATIVE AND FORMATIVE ASSESSMENTS TELL THE SAME STORY? COMPARING RESIDENT PERFORMANCE IN A SURGICAL FOUNDATIONS BOOT CAMP.
N Wagner, BSc, N Amin, MD, FRCS, S Kelly, MD, FRCS, R R Sonnadara, PhD; McMaster University

Introduction: As programs shift towards competency-based medical education (CBME), priority has been placed on increasing the number of low-stakes, formative assessments (RCPSC, 2011). While this aligns with theoretical frameworks suggesting more frequent, formative, evaluations are likely a better indicator of resident performance than high-stakes summative evaluations (van der Vleuten et al., 2012), little evidence on the correlation between formative and summative assessment scores exists in surgical training. The purpose of this study was to compare resident performance on a summative objective structured clinical exam (OSCE) with multiple, formative workplace based assessments completed at the onset of residency training.

Methods: Thirty-six first year residents from McMaster University participated in a summative OSCE at the end of a Surgical Foundations boot camp. The OSCE included stations on informed consent, suturing, breaking bad news, foley catheter insertion, and hand ties. Following the boot camp, residents were asked to complete a six formative workplace-based assessments on at least two of the five aforementioned skills. As residents typically completed multiple workplace-based assessments on two or three of the five skills, we were unable to perform one analysis across all skills. Instead, OSCE scores were compared with workplace-based assessment scores on each task using a separate paired t test.

Results: Our results suggest there was no significant difference between workplace-based assessment and OSCE scores for suturing (t(29)= 1.21; p = 0.24) and foley catheter insertion (t(11)=1.57; p = 0.14). However, workplace-based assessment scores were significantly higher than OSCE scores for informed consent (t(20)= 6.85; p<0.01), breaking bad news (t(10)= 5.88; p<0.01), and hand ties (t(26)= 2.18; p<0.05).

Conclusion: The results from this study suggest that our formative and summative assessments may capture different aspects of resident performance depending on the skill being evaluated. For some skills, performance in the OSCE under predicted actual clinical performance. Further work needs to be done to investigate why these differences exist and whether they are mediated by factors such as type of skill (technical versus nontechnical), stress, time to complete task, and rater.