PS4 - 02: WHAT DO QUANTITATIVE RATINGS AND QUALITATIVE COMMENTS TELL US ABOUT GENERAL SURGERY RESIDENTS’ PROGRESS TOWARD INDEPENDENT PRACTICE? EVIDENCE FROM A 5-YEAR LONGITUDINAL COHORT
Ara Tekian, PhD, MHPE, Martin Borhani, MD, Eric Abasolo, BS, Sarette Tilton, BS, Yoon Soo Park, PhD; University of Illinois at Chicago

Background: Faculty use assessment data consisting of quantitative ratings and qualitative comments to provide feedback to learners and to inform the clinical competency committee (CCC) on progress toward independent practice. However, it is unclear how quantitative and qualitative information are synthesized and linked with milestones-based feedback. This study examines the alignment of quantitative and qualitative assessment data using a longitudinal cohort of residents progressing throughout the five-year general surgery residency.

Methods: Rotation evaluation data were extracted for 171 residents who trained between July 2011 and July 2016. Data included 6,069 rotation evaluations forms completed by 38 faculty members and 164 peer-residents. Descriptive statistics and mixed-effects regression were used to examine trends in quantitative evaluation scores; generalizability theory was used to estimate variance components and reliability. Qualitative comments mapped to general surgery milestones were coded for positive/negative feedback and quality/relevance. Themes were identified using constant comparative analysis associated with constructivist grounded theory.

Results: Evaluation scores increased significantly across the five years, p = .010. Milestones targeting Patient Care, Medical Knowledge, and System-Based Practice had the largest improvement over time. The reliability of rotation evaluations ratings was good, Φ-coefficient = .72. Quantitative evaluation scores were significantly correlated with positive/negative feedback indicating alignment between quantitative and qualitative feedback, r = .52. However, when residents received higher quantitative ratings, the quality of comment was significantly lower, r = –.20. Themes identified from qualitative comments included feedback on leadership, teaching contribution, medical knowledge, work ethic, patient-care, and ability to work in a team-based setting. As residents progressed in training, faculty comments became more critical, highlighting deficiencies; however, most comments were positive and predicted success by the fifth year. Faculty comments focused on technical and clinical abilities; comments from peers focused on professionalism and interpersonal relationships.

Conclusions: While there was alignment between quantitative and qualitative assessment data, we found differences in the degree of alignment and types of themes emphasized as residents progressed in their surgical training. These findings underscore the need to improve our understanding of how faculty synthesize assessment data and the types of competency-specific feedback they provide as residents progress toward independent practice.