How many local patients does a hospital need before switching on an imported sepsis prediction model? A two-system evaluation of a silent-mode acceptance and local adaptation protocol

Objective: Hospitals, including most in the Arab region, switch on imported prediction models with a vendor threshold and a performance claim measured elsewhere. We quantified how many local patients an adopting hospital needs to detect a broken claim and to adapt the model, and the alert burden of each option.

Methods: We used 40,336 intensive care unit admissions from two independent hospital systems (PhysioNet/CinC Challenge 2019; Sepsis-3 labels). A gradient-boosting early-warning model developed at one system, with a threshold for 80% sensitivity, was transported to the other, in both directions. With 100–10,000 local admissions, we evaluated silent-mode acceptance testing, threshold reset with and without labels, logistic recalibration, local retraining and vendor-informed local updating on a fixed local test set.

Results: Transported unchanged, patient-level AUROC fell from 0.830 to 0.738 and from 0.834 to 0.726. The failures differed: sensitivity fell to 56.9% in one direction, while in the other alert episodes rose from 41 to 89 per 100 patient-days. Silent mode detected the sensitivity shortfall in 90% of repeats with 500 admissions and 95% with 1,000; the alert-burden rise was visible without labels at 100 admissions. A labelled threshold reset reached 80±5% sensitivity in 80–85% of repeats at 1,000 admissions and in all repeats at 2,000. Local models beat the imported model in 85–90% of repeats at 1,000 admissions and in all from 2,000, reaching AUROC 0.847 and 0.815 with fewer alerts.

Conclusions: About 1,000 labelled local admissions should be a minimum before go-live, and procurement contracts should secure the right to retune and locally update the model.

Author: Ahmed Abdelalim, BPharm (Al-Azhar University), VisionSeek, Incheon, South Korea.

Preprint, not peer reviewed. Files: manuscript PDF, supplementary files and a code/derived-results archive. Raw data: PhysioNet/CinC Challenge 2019 (https://doi.org/10.13026/v64v-d857, CC BY 4.0), downloaded by the code and not redistributed.