Jev in Medicine: A Benchmark Evaluation

Background: Jev is a non-generative System One model that assigns probabilities to predefined answer options and cannot answer outside them. Its calibration and its performance on complex diagnostic cases remain insufficiently characterised.

Methods: We evaluated Jev 1.13 on four medical benchmarks: MetaMedQA (1,373 examination questions), PubMedQA (500 research questions), DiagnosisArena-MCQ (915 cases) and the NEJM Case Challenges (34 cases). GPT-6 Sol, with and without reasoning, was the reference. Outcomes were top-1 accuracy (primary), calibration, selective prediction and recognition of unanswerable questions

Results: Jev's accuracy was similar to that of GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%), lower on MetaMedQA (74.8% vs 82.7%) and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%) and the NEJM cases (61.8% vs 82.4%, adjusted P = 0.078). On MetaMedQA, Jev's probabilities were the best calibrated (expected calibration error 0.063 vs 0.146), and its answers with a probability ≥0.9 (52.9% of questions) were 93.4% accurate, but GPT-6 Sol was as accurate when it accepted a similar proportion of questions. On DiagnosisArena-MCQ, Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768). Of the 162 questions whose correct answer was "I don't know or cannot answer", Jev chose that option for 10.5% (GPT-6 Sol, 8.6%). Median latency was 0.27–0.31 s; all 2,823 items cost US$0.08.

Conclusions: Jev was fast and inexpensive, and its accuracy was similar to that of a frontier LLM on research abstracts but lower on examination questions and much lower on complex diagnostic cases. Its better calibration on examination questions gave no selective-prediction advantage, and it seldom chose "I don't know" when that was the correct answer. Task-specific validation is required before clinical use.