Surgical Scene Understanding using Context Prompting for Multimodal Large Language Models

Surgical video analysis requires domain-specific expertise and
knowledge extraction. Artificial Intelligence (AI) approaches for surgical
video analysis rely on extensive computational resources, large
annotated datasets, and specialised architectures while offering limited
explainability. This work explores the potential of Multimodal Large
Language Models (MLLMs) for surgical video analysis. Specifically, we
develop an interactive framework to use the Gemini API to identify
instrument-verb-target triplets in laparoscopic cholecystectomy videos.
Our approach implements a dual-context prompt engineering approach
that comprises a surgical context prompt (SCP) and a surgical analysis
prompt (SAP) that separates domain context knowledge from the analytical
task. The dual-context prompts utilised XML-structured knowledge
representation with distinct SCP and SAP. Evaluation on the validation
set of CholecT50 (laparoscopic cholecystectomy videos) dataset demonstrates
the potential of MLLM to analyze surgical frames and identify the
instrument-verb-target triplets. However, the accuracy measure is lower
than that of purpose-built models. This work establishes a valuable foundation
for MLLM applications in surgical education and research, with
potential to evolve and improve with our ongoing work in this direction.