PhaKIR Dataset - Surgical Procedure Phase, Keypoint, and Instrument Recognition

To access the dataset, the following steps have to be performed:

1. Register on the Zenodo platform and login: Only then the Access Request Form below will become visible.
2. Fill out and submit the Request Access Form appearing below.
3. The access request will be verified and you will be notified by email as soon as access is granted. 

The dataset is described in the following publications: 

• Rueckert, Tobias et al.: Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge. arXiv preprint, https://arxiv.org/abs/2507.16559. 2025.
• Rueckert, Tobias et al.: Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR). arXiv preprint, https://arxiv.org/abs/2511.06549. 2025.

All resulting works using this dataset in part or in whole must cite the following publications:

@article{rueckert2025comparative,
  author       = {Tobias Rueckert and David Rauber and Raphaela Maerkl and Leonard Klausmann and Suemeyye R. Yildiran and Max Gutbrod and Danilo Weber Nunes and Alvaro Fernandez Moreno and Imanol Luengo and Danail Stoyanov and Nicolas Toussaint and Enki Cho and Hyeon Bae Kim and Oh Sung Choo and Ka Young Kim and Seong Tae Kim and Gon{\c{c}}alo Arantes and Kehan Song and Jianjun Zhu and Junchen Xiong and Tingyi Lin and Shunsuke Kikuchi and Hiroki Matsuzaki and Atsushi Kouno and Jo{\~{a}}o Renato Ribeiro Manesco and Jo{\~{a}}o Paulo Papa and Tae{-}Min Choi and Tae Kyeong Jeong and Juyoun Park and Oluwatosin Alabi and Meng Wei and Tom Vercauteren and Runzhi Wu and Mengya Xu and An Wang and Long Bai and Hongliang Ren and Amine Yamlahi and Jakob Hennighausen and Lena Maier{-}Hein and Satoshi Kondo and Satoshi Kasai and Kousuke Hirasawa and Shu Yang and Yihui Wang and Hao Chen and Santiago Rodr{\'{\i}}guez and Nicol{\'{a}}s Aparicio and Leonardo Manrique and Juan Camilo Lyons and Olivia Hosie and Nicol{\'{a}}s Ayobi and Pablo Arbel{\'{a}}ez and Yiping Li and Yasmina Al Khalil and Sahar Nasirihaghighi and Stefanie Speidel and Daniel Rueckert and Hubertus Feussner and Dirk Wilhelm and Christoph Palm},
  title        = {{Comparative} validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: {Results} of the {PhaKIR} 2024 challenge},
  journal      = {CoRR},
  volume       = {abs/2507.16559},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2507.16559},
  doi          = {10.48550/arXiv.2507.16559}
}

@article{rueckert2025video,
  author       = {Tobias Rueckert and Raphaela Maerkl and David Rauber and Leonard Klausmann and Max Gutbrod and Daniel Rueckert and Hubertus Feussner and Dirk Wilhelm and Christoph Palm},
  title        = {{Video} {Dataset} for {Surgical} {Phase}, {Keypoint}, and {Instrument} {Recognition} in {Laparoscopic} {Surgery} {(PhaKIR)}},
  journal      = {CoRR},
  volume       = {abs/2511.06549},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2511.06549},
  doi          = {10.48550/arXiv.2511.06549}
}

@article{wagner2023comparative,
  author       = {Martin Wagner and Beat P. M{\"{u}}ller{-}Stich and Anna Kisilenko and Duc Tran and Patrick Heger and Lars M{\"{u}}ndermann and David M. Lubotsky and Benjamin M{\"{u}}ller and Tornike Davitashvili and Manuela Capek and Annika Reinke and Carissa Reid and Tong Yu and Armine Vardazaryan and Chinedu Innocent Nwoye and Nicolas Padoy and Xinyang Liu and Eung{-}Joo Lee and Constantin Disch and Hans Meine and Tong Xia and Fucang Jia and Satoshi Kondo and Wolfgang Reiter and Yueming Jin and Yonghao Long and Meirui Jiang and Qi Dou and Pheng{-}Ann Heng and Isabell Twick and Kadir Kirta{\c{c}} and Enes Hosgor and Jon Lindstr{\"{o}}m Bolmgren and Michael Stenzel and Bj{\"{o}}rn von Siemens and Long Zhao and Zhenxiao Ge and Haiming Sun and Di Xie and Mengqi Guo and Daochang Liu and Hannes G{\"{o}}tz Kenngott and Felix Nickel and Moritz von Frankenberg and Franziska Mathis{-}Ullrich and Annette Kopp{-}Schneider and Lena Maier{-}Hein and Stefanie Speidel and Sebastian Bodenstedt},
  title        = {{Comparative} validation of machine learning algorithms for surgical workflow and skill analysis with the {HeiChole} benchmark},
  journal      = {Medical Image Analysis},
  volume       = {86},
  pages        = {102770},
  year         = {2023},
  url          = {https://doi.org/10.1016/j.media.2023.102770},
  doi          = {10.1016/J.MEDIA.2023.102770}
}

The proposed dataset was used as the training dataset in the PhaKIR challenge (https://phakir.re-mic.de/) as part of EndoVis-2024 at MICCAI 2024 and consists of eight real-world videos of human cholecystectomies ranging from 23 to 60 minutes in duration. The procedures were performed by experienced physicians, and the videos were recorded in three hospitals. In addition to existing datasets, our annotations provide pixel-wise instance segmentation masks of surgical instruments for a total of 19 categories, coordinates of relevant instrument keypoints (instrument tip(s), shaft-tip transition, shaft), both at an interval of one frame per second, and specifications regarding the intervention phases for a total of eight different phase categories for each individual frame in one dataset and thus comprehensively cover instrument localization and the context of the operation. Furthermore, the provision of the complete video sequences offers the opportunity to include the temporal information regarding the respective tasks and thus further optimize the resulting methods and outcomes.