Poster1-08: REALIZING THE FORMATIVE POTENTIAL OF A SURGICAL EDUCATION EXAMINATION: MOVING BEYOND TEST ADMINISTRATION
Maryam Wagner, PhD¹, Carlos Gomez-Garibello, PhD¹, Paola Fata, MDCM, FRCSC¹, Brock Vair²; ¹McGill University, ²Dalhausie University

Introduction: Formative assessments are a powerful mode of advancing learning by providing learners with feedback that highlights strengths and identifies areas for improvement. The information generated from these tests also can have significant impact on the development, design, and delivery of the surgical education programs. Advancing these goals requires purposeful test design, and examination of multiple sources of evidence generated from its use. The purpose of this study was to examine the extent to which a nationally administered exam of general surgical knowledge achieved its formative potential.

Methods: A mixed methods approach was used to conduct the investigation. General Surgery residents (N = 453) completed a formative test examining different domains.  The exam comprised 143 questions examining foundational and core knowledge. In addition, all examinees provided their perceptions about the educational value of the exam using a survey immediately after completing the test. Further information was gathered from a subset (n= 24) of residents one year after taking the test. Residents shared their perceptions of the exam during a learning session in which exam questions, their design, and responses were discussed. The residents’ also discussed the extent to which the exam contributed to their education, and their opinions on how the exam could be improved.

Results: Analysis of test performance data, survey results and qualitative data from the learning session generated evidence about the ways in which the exam achieved its formative potential for different stakeholders, and for different purposes. Test results provided information about learners’ performance based on their level of training, and across the different types of knowledge tested. The integration of these findings alongside residents’ feedback provided program directors with information about gaps in the curriculum, and critical periods (PGY4) in which learning faltered. Residents valued the test, and used their individual performance data (differentiated by types of knowledge tested) to guide their studies. Residents’ feedback also informed the extent to which the formative exam aligned with curricular goals and education.

Conclusions: This test highlights the successful design and implementation of a formative exam to advance surgical education at multiple levels: residents, program directors, and surgical program.