IgakuQA119: LLM Evaluation on the 119th Japanese Medical Licensing Examination

v1.0.0: Initial Public Release

This is the first public release of IgakuQA119, a comprehensive framework for evaluating Large Language Models (LLMs) on the 119th Japanese Medical Licensing Examination (JMLE).

This project, inspired by the nmle-rta repository, provides a complete and reproducible workflow for assessing the capabilities of modern LLMs in a specialized, high-stakes domain.

Key Features in this Release:

• Comprehensive Evaluation Dataset: Includes the full question set from the 119th JMLE.
• Flexible LLM Support: Natively supports major cloud APIs (OpenAI, Anthropic, Gemini, OpenRouter) and local LLMs via Ollama.
• Streamlined YAML-based Workflow: A new, unified run_exp.sh script, controlled by a central experiments.yaml file, manages the entire lifecycle of an evaluation: setup, execution, re-running skipped questions, and grading.
• Automated Leaderboard: The grade task automatically calculates scores and updates the leaderboard in the README.md.
• Transparent Data Provenance: Includes details and scripts related to dataset acquisition and preprocessing, ensuring transparency and reproducibility.

Dataset Source:

The question data (text, choices, images) was processed from official JMLE PDFs via OCR by the author of the nmle-rta project, with permission obtained for its use. The grading logic is based on official information from Japan's Ministry of Health, Labour and Welfare (MHLW).

For detailed setup and usage instructions, please refer to the main README.md file.

This project is licensed under the Apache License 2.0.