case_images: PMC11814077_01
case_images: Exemplary LLM query and analysis (Eurorad case ID 12746).. Mistral-Small was instructed to determine the most likely differential diagnoses based on a textual case description that included a condensed medical history and imaging findings. The LLM output contained three diagnoses with a rationale for each suggestion. Llama-3-70B was subsequently utilized to classify the response as correct (if the true diagnosis is included in the suggestions) or wrong. The true diagnosis in this case was 'jugular foramen meningioma'.
case_images: 41746_2025_1488_Fig5_HTML.jpg
case_images: PMC11814077_01_41746_2025_1488_Fig5_HTML.jpg
case_images: Fig5
case_images: Using a total of 140 LLM responses for which both human "ground truth" and LLM judge assessments were known, we calculated the accuracy of the LLM judge (Fig. 5).