PS4-10: STRUCTURING EXPERT INSIGHTS FOR AUTOMATED COACHING OF SURGICAL SKILLS
Emily Bontrager, S. Swaroop Vedula, MBBS, PhD, Michael Peven, MSE, Christina Harnett, PhD, Lieny Jeon, PhD, Linda Tsantis, EdD, Bethany C Sacks, MD, MEd, Gina L Adrales, MD, MPH, Gregory D Hager, PhD, Anand Malpani, PhD; Johns Hopkins University

INTRODUCTION: Surgical educators are able to deconstruct performance and provide feedback on targeted activities. While assessments may be captured in structured rating scales, there are limited tools to capture feedback. We propose to use “think aloud” (TA) and “video commentary” (VC) to capture expert insights on surgical skills teaching and feedback through narratives. Our overall goal is to use such narratives to automate coaching of surgical skills. Here, we present a qualitative data analysis of expert surgeon narratives.

METHODS: We collected narrations from expert surgeons using needle passing tasks in the da Vinci Skills Simulator in two settings: 1) TA: surgeon performs task while thinking out loud about what-how-why of the performance (what are they doing, how are they doing, and why are they doing it in a particular manner), 2) VC: surgeon watches previously recorded performances and narrates their thoughts (what-how-why format). We debriefed the narrator at the end to clarify any jargon. Two individuals independently transcribed narratives using YouTube for automatic transcriptions. We used a qualitative data analysis software, Dedoose, to mark excerpts that contained comments for teaching and feedback. These excerpts were assigned codes for “activity execution (AE)” (how a task was/should be performed) and “value” (why an action is consequential). Finally, we mapped each excerpt to specific activities, e.g., “grasping needle”, “rotating needle.” We analyzed frequency of AE and value excerpts across activities for TA and VC.

RESULTS: We analyzed eight transcripts (four TAs and four VCs) obtained from two surgeons. We marked 188 excerpts (106 within TA, 82 within VC). The frequency of codes assigned was: AE (113, 67, 46) and value (75, 39, 36) for total count, within TA and within VC, respectively. We observed that most AE and value excerpts referred to“grasping needle”, followed by “rotating needle”  (ref. Figure).  

CONCLUSIONS: Think aloud and video commentary can be used to capture expert insights on surgical skills teaching and feedback. With a representative expert cohort, it is possible to reach consensus on the specific feedback and content to be provided to a trainee that can support a automated virtual coach.