Inter-Rater Reliability of a Suture Assessment Tool

Introduction: Despite significant advancements in standardizing medical knowledge assessment, evaluating technical skills, such as suturing, has largely depended on subjective faculty evaluations, creating a gap in reliable, objective assessment. Addressing this need, Sundhagen et al. developed a suturing assessment tool, demonstrating promising reliability and validity in standardized settings with medical students. However, its performance remains untested when used by family medicine faculty to evaluate residents in a typical clinical environment. This pilot study aims to assess the inter-rater reliability of the modified Sundhagen tool in evaluating suturing skills among family medicine residents in real-world settings.

Methods: To assess inter-rater reliability, a modified version of the Sundhagen assessment tool, comprising eight yes-or-no questions on suturing skills, was provided to four family medicine faculty members in La Crosse, WI. Faculty members used this tool to evaluate five videos of family medicine residents performing simple sutures on chicken breast incisions, captured during routine educational activities without specific training on tool use. The videos, obtained from a prior residency study with participant consent, displayed only the residents’ hands and suturing materials. Task completion scores were tallied, and inter-rater reliability was measured using ICC, with analysis by question.

Results: The ICC score was 0 (95% confidence interval -0.18 – 0.61), suggesting poor inter-rater reliability.

Conclusion: This pilot study did not demonstrate strong inter-rater reliability for the modified Sundhagen tool when applied to family medicine residents, highlighting the need for further refinement and assessment of additional validity components. As an essential preliminary step, these findings lay the groundwork for a larger, adequately powered trial that can more rigorously evaluate the tool's performance across a broader set of metrics, enhancing its applicability and reliability in family medicine residency training.