Public repository for "A Real-Time Artificial Intelligence Diagnostic Copilot in Simulated Primary Care Consultations: Randomized Simulation Study"

This repository contains the complete reproducibility package supporting the manuscript “A Real-Time Artificial Intelligence Diagnostic Copilot in Simulated Primary Care Consultations: Randomized Simulation Study.”

The study was a formative, case-level randomized, adjudicator-blinded, simulation-based physician-performance study conducted in a secure web-based virtual primary care clinic. Thirteen board-certified family and community medicine physicians completed 260 voice-based simulated consultations based on 40 diagnostically challenging adult clinical vignettes. Each consultation was assigned to either real-time assistance from the Medsys AI diagnostic copilot or an unassisted, resource-restricted simulation condition without external diagnostic aids. No real patients were enrolled, no clinical care was delivered or modified, and no identifiable patient health data are included in this repository.

The repository brings together the study protocol, clinical case materials, blinded adjudication documentation, the episode-level analysis dataset, inter-rater agreement tables, an expanded data dictionary, the executable statistical-analysis script, complete console outputs from the full and simulator-exclusion analyses, an analytical transparency and reproducibility document, materials documenting the construction and post hoc evaluation of the virtual-patient simulator, and the formal institutional determination regarding ethics committee approval.

The final statistical analysis differs from the initial statistical-analysis plan described in the original protocol. The final analyses use frequentist binomial generalized linear mixed models (GLMMs) fitted by maximum likelihood, with crossed random intercepts for physician and case where applicable. Python 3.11 is used for data processing and orchestration, while the GLMMs are fitted in R 4.6.1 using lme4 2.0-6, invoked noninteractively from Python through Rscript. The principal inferential workflow uses likelihood-ratio tests for fixed-effect contrasts, profile-likelihood confidence intervals for odds ratios, Gauss-Hermite integration for population-averaged marginal quantities, and 10,000-replicate parametric bootstrap procedures for confidence intervals of marginal probabilities and derived measures. These methods, together with the sensitivity, safety, workflow, moderation, and exploratory analyses, are documented in the public analysis script, its complete console outputs, and the analytical transparency document.

Following completion of the study, the study documentation was reviewed by the Research Ethics Committee for Medicinal Products (CEIm) of Hospital Universitario La Paz, Madrid, Spain. The Committee issued a formal institutional determination that approval by a CEI/CEIm was not required because the study did not involve personal health data or biological samples from participants.

What is included

1. Study protocol

• PC-MEDSYS protocol_V1.1 Feb 2025.pdf

Full original study protocol describing the study design, eligibility criteria, randomization, masking, procedures, outcomes, adjudication process, sample-size assumptions, and initially planned statistical analysis.

The protocol is provided in its original form. Subsequent analytical refinements and post hoc analyses are documented transparently in the final statistical-analysis script and the analytical transparency document.

2. Clinical case materials

• Details on 40 cases used in the trial.pdf

Description of the 40 adult clinical vignettes used to construct the virtual-patient simulations, including case identifiers, gold-standard diagnoses, demographics, original clinical content, and simulator-related case information.

3. Blinded adjudication package

• Adjudicator 1-Fernando Cerro Zarabozo.pdf

• Adjudicator 2-Sara Garcia de Francisco.pdf

• Adjudicator 3-FERNANDA DOMINGUEZ.pdf

Row-level documentation of the independent blinded adjudication of physician and AI-generated diagnostic outputs. Diagnostic correctness was determined using the predefined acceptance criteria described in the study protocol.

4. Data dictionary

• Updated_Data_Dictionary_Randomized_Simulation_Study_v2_20260820.docx

Expanded data dictionary describing the main dataset and its relationship to the adjudication files. It documents source variables, identifiers, trial-arm coding, diagnostic outputs, timing variables, adjudicator-derived labels, Top-1/Top-2/Top-3 outcome derivation, safety-analysis variables, derived analysis variables, simulator-exclusion identifiers, join rules, and integrity checks used for reproducible analysis.

5. Primary dataset and inter-rater agreement tables

• Phase_3_complete results_23092025.csv

Main episode-level analysis dataset containing one row per randomized simulated consultation (n=260). This is the principal input dataset used by the statistical-analysis script.

• Phase_3_complete results_12062025 - Concordancia evaluadores para diagnóstico médico.csv

Three-rater adjudication table used to calculate inter-rater agreement for physician-submitted diagnoses.

• Phase_3_complete results_12062025 - Concordancia evaluadores para diagnóstico Medsys.csv

Three-rater adjudication table used to calculate inter-rater agreement for AI-generated diagnoses.

6. Statistical-analysis code and complete outputs

• Data processing script-In silico trial with Medsys AI_updated_19082026.py

End-to-end executable Python analysis script. Python performs data loading, validation, deterministic endpoint derivation, descriptive analyses, inter-rater reliability calculations, and orchestration of the statistical models. Frequentist GLMMs are fitted in R 4.6.1 with lme4 2.0-6 through noninteractive calls to Rscript.

The script includes:

• Fleiss’ κ inter-rater reliability analyses;

• the primary Top-3 diagnostic-accuracy GLMM;

• likelihood-ratio inference and profile-likelihood odds-ratio intervals;

• Gauss-Hermite marginalization;

• 10,000-replicate parametric-bootstrap confidence intervals;

• adjusted probabilities, absolute risk differences, risk ratios, relative error-rate reductions, and simulation-context NNT-equivalents;

• leave-one-physician-out sensitivity analyses;

• post hoc Top-1 and Top-2 analyses;

• consultation-duration analyses;

• exploratory case-difficulty and physician-ability moderation analyses;

• recent-workload and consultation-order analyses;

• the secondary safety analysis of concordance with incorrect operative AI suggestions;

• adjustment for consultation duration; and

• exploratory physician-versus-standalone-AI comparisons.

The same script contains a predefined switch for reproducing either the complete n=260 analysis or the post hoc simulator-exclusion sensitivity analysis with the 16 flagged consultations removed.

• Phase_3_postprocessed_results_all_cases (n=260)_10000_bootstrap_reps.txt

Complete, unedited console output from the canonical full-dataset analysis of all 260 randomized consultations using 10,000 parametric-bootstrap replicates. It contains the numerical results underlying the manuscript, including model coefficients, profile-likelihood intervals, likelihood-ratio tests, random-effect variance components, bootstrap marginal estimates and confidence intervals, model diagnostics, sensitivity analyses, and exploratory analyses.

• Phase_3_postprocessed_results_good_cases (n=244)_10000_bootstrap_reps.txt

Complete console output from the post hoc virtual-patient simulator-exclusion sensitivity analysis. The same statistical workflow is repeated after excluding the 16 consultations flagged by the retrospective simulator-performance audit, leaving n=244 consultations. No model specification or inferential setting is changed by this exclusion.

7. Analytical transparency and reproducibility document

• Study_Analytical_Transparency_Document_Frequentist_GLMM_v2_20260820.docx

Detailed end-to-end explanation of the final statistical workflow. The document describes the provenance and transformation of the analysis variables, the Python-to-R execution architecture, the frequentist mixed-model specifications, likelihood-based inference, marginalization and bootstrap procedures, reproducibility settings, sensitivity analyses, and the correspondence between canonical console outputs and the values reported in the manuscript.

It is intended to facilitate independent statistical verification without requiring the reader to infer the analytical workflow solely from the source code.

8. Virtual-patient construction and post hoc performance evaluation

• patient_adherence_evaluation_updated05072026.ipynb

Executable Jupyter notebook implementing the retrospective post-trial quality-assurance audit of virtual-patient behavior. It evaluates the transcripts of the 260 simulated consultations across predefined performance domains, including diagnostic leakage, excessive helpfulness, conversational fluency or blocking, and adherence to the required physical-examination response format.

• Virtual_patient_performance_results.txt

Complete output from the retrospective virtual-patient performance audit, including row-level evaluator rationales and categorical classifications for all 260 interactions. These results identify the consultations subsequently flagged for the simulator-exclusion sensitivity analysis.

• Virtual patient construction and post hoc analysis of performance.pdf

Companion technical document describing the construction of the virtual-patient simulator and its retrospective performance evaluation. It summarizes the simulator architecture and prompting approach, the conversational and physical-examination constraints, the post hoc audit methodology, observed simulator deviations, physician feedback relevant to simulator performance, and the comparison between the full n=260 analysis and the n=244 simulator-exclusion sensitivity analysis.

• Phase_3_postprocessed_results_good_cases (n=244)_10000_bootstrap_reps.txt

The statistical-output file described above also serves as the complete computational record of the simulator-exclusion robustness analysis.

9. Institutional ethics determination

• Certificado PI 2026.715_62298015.pdf

Formal certificate issued by the Research Ethics Committee for Medicinal Products (CEIm) of Hospital Universitario La Paz following review of the study documentation.

The certificate identifies the study as 2026.715 and states that, because the initiative did not involve personal health data or biological samples from participants, approval by a CEI/CEIm was not required. The determination was recorded in Act No. 15/2026 and issued in Madrid on August 11, 2026 (Reference 57/230428.9/26).

Reproducing the statistical analyses

To reproduce the manuscript analyses, place the main episode-level dataset and the two adjudication CSV files in the same directory as the statistical-analysis script and execute:

Data processing script-In silico trial with Medsys AI_updated_19082026.py

The canonical analysis environment uses:

• Python 3.11

• pandas 2.2.3

• statsmodels 0.14.4

• SciPy 1.15.2

• R 4.6.1

• lme4 2.0-6

R must be installed and accessible to the Python script through Rscript. The final canonical results use 10,000 parametric-bootstrap replicates. The complete reference outputs for both the n=260 primary analysis and the n=244 simulator-exclusion sensitivity analysis are included in the repository for direct comparison with an independent reproduction.

This repository is intended to support peer review, independent statistical re-analysis, methodological scrutiny, and transparent reuse of the study materials. The combination of the public episode-level dataset, blinded adjudicator-derived outcomes, executable analysis code, canonical console outputs, expanded data dictionary, and analytical transparency document permits end-to-end third-party verification of the statistical findings reported in the manuscript.