Published August 31, 2026 | Version v0.4.13

EleutherAI/lm-evaluation-harness: v0.4.13

Description

v0.4.13 Release Notes

A fix-focused release. The main fixes are for few-shot leakage, a multiple-choice filter bug, and group stderr, alongside two new ONNX backends and eight new benchmark suites. Also updated most configs for datasets>=4, which accounts for much of the diff by volume.

Highlights

Bug Fixes

Fixes that may shift previously reported numbers:

  • Eval documents leaked into few-shot prompts. The sampler could draw the document under test into its own demonstrations by @SiavashShams in #3978, and gen_prefix was resolved against the eval doc rather than the few-shot doc — splicing the evaluated question into every shot for RULER niah_single_1, humaneval_instruct, and humaneval_64_instruct by @adityasingh2400 in #3979
  • MultiChoiceRegexFilter prefix-shadowing. Regex alternation is leftmost-wins, so a choice that prefixed a longer choice ("Guilty" vs "Guilty of Romance") matched inside it and scored correct answers as wrong — visible on BBH movie_recommendation by @iamsharduld in #3884
  • Group stderr with weight_by_size: false. Groups reported the size-weighted pooled stderr even when the point estimate was an unweighted mean, giving error bars up to ~3x too narrow. Only unequal-sized subtasks change by @iamsharduld in #3882
  • minerva_math answer normalization. sqrt shorthand no longer corrupts indexed roots (#4037), the thousands-separator strip no longer fuses digit tuples (0,1 → 01, #4039), an answer identical to the gold now scores correct (#4034), and the few-shot prompt LaTeX is corrected (#4045) by @feiiiiii5 and @nata2627. putnam_axiom shares these helpers and picks up the same fixes.

New Model Backends

  • onnxruntime — raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in #3984
  • onnxruntime-genai — cross-platform ONNX Runtime GenAI backend, with winml refactored on top of it by @thiagocrepaldi in #3960
  • Megatron-LM v0.18 compatibility by @oberpierre in #3991

Install with pip install lm_eval[onnxruntime] or lm_eval[onnxruntime-genai]. For non-CPU execution providers install the matching wheel instead — onnxruntime-gpu / onnxruntime-rocm, or onnxruntime-genai-cuda / onnxruntime-genai-directml — these are mutually exclusive.

New Tasks

  • LongProc — long-context procedural reasoning, 6 task types across 16 configs by @xiye17 in #3544
  • Uncheatable Eval — contamination-resistant evaluation: rolling log-likelihood over recently-published documents from Wikipedia, GitHub, BBC News, arXiv, bioRxiv, and AO3, as 15 category tasks plus a size-weighted group by @ziqing-huang in #3442
  • TyDiQA Gold Passage — 9-language multilingual QA by @bongho in #4044
  • LegalBench — new suite, 19 tasks: the HELM-lite LegalBench subset (5 tasks, #3860) and the Contract NLI suite (14 NDA entailment tasks, #3954) by @bongho
  • IndicParam — MCQ benchmark covering 12 low-resource Indic languages by @hjoshifonteva in #3826
  • IndicXNLI Gujarati (indicxnli_gu) by @bhaumik611 in #4056
  • GreekMMLU — official native-sourced configuration by @mersinkonomi in #3581
  • Physics GRE — InflectionAI multiple-choice physics benchmark by @bongho in #3853
  • Putnam Axiom — competition-level mathematical reasoning by @baberabb in #3998
  • Portuguese Bench — ASSIN2 RTE and STS by @bongho in #3812
  • Catalanbench — cieaCOVA (#4043) and Terretaqa (#3784) by @baberabb and @ivanmartinezmurillo
  • MLQA — a tag to run all variants at once by @ivanbaldo in #2977

Task Changes

Correctness & Prompts

  • global_piqa restructured into parallel/non-parallel × cloze/generation variants; the global_piqa_completions and global_piqa_prompted groups are now global_piqa_cloze and global_piqa_generation by @baberabb in #3816
  • IrokoBench — repaired afrimmlu / afrimgsm / afrixnli task registration by @discobot in #3841, fixed the broken afrisenti / mafand prompt_2 group references by @DaoyuanLi2816 in #3847, and switched afrixnli prompt_1 doc_to_text to Jinja braces by @Solaris-star in #3944
  • KorMedMCQA — answer-choice extraction now follows the paper's Appendix B by @discobot in #3842
  • JSONSchema Bench — per-sample validation timeout so a pathological schema no longer hangs the whole eval by @hancheolcho in #3923
  • RULER — every task crashed with TypeError: unhashable type: 'dict' before inference when the tokenizer arrived as anything but a plain string (e.g. under local-chat-completions); the name is now resolved before the cached lookup by @nata2627 in #4048
  • NIAH — no longer loops forever when max_seq_lengths < 4096 by @vnayakde in #3372
  • LongBench — code_sim_score skips a blank leading line before extraction by @cameronshinn in #3921
  • FDA / SWDE / SQuAD_completion — whitespace stripped from input and target by @EphraiemSarabamoun in #3795
  • med_prescriptions — requires both keys before combining complaints and diagnosis by @Anai-Guo in #4041
  • HumanEval — humaneval_random_span_infilling_light was registered under the wrong task name, causing a KeyError by @jaydeepborkar in #3768
  • INCLUDE — stray space removed from North Macedonian task identifiers by @DaoyuanLi2816 in #3848
  • DarijaBench — trasnlation typo corrected in translation task names by @DaoyuanLi2816 in #3824
  • Unitxt — dataset_kwargs is now passed through (#3230) and init sets the task name (#3225) by @mprahl
  • format_span filter — normalizes labels only, leaving entity text untouched by @k-dickinson in #3887
  • ContextSampler — no longer crashes when the few-shot pool contains duplicate eval-doc rows by @Kymi808 in #3790
  • fewshot_config.split — a nested fewshot_config.split now takes precedence over the inherited top-level fewshot_split, as documented by @chuenchen309 in #3937

Model Backends & APIs

  • Chat completions now support think_end_token by @yaodong-shen in #3959
  • auto:N batch size is parsed correctly in API models by @AbdullahRasheed45 in #3970, and batch_size="auto" no longer raises on the neuronx backend by @AbdullahRasheed45 in #3971
  • Synchronous API requests honor the configured timeout by @SiavashShams in #3995
  • Anthropic stop sequences are kept non-empty by @he-yufeng in #3822
  • HF: max_length is detected from a nested text_config (Gemma3 multimodal) by @OrionArchitekton in #3916, max_cpu_memory is passed through to accelerate's max_memory by @Anai-Guo in #4016, and tokenizers that decode to an empty string fall back to tokenizer.eos_token by @ganeshr10 in #3657
  • vLLM warns when it ignores the device argument by @Apeironics in #3803; sglang argument passing fixed by @wm901115nwpu in #3817

Core, CLI & Config

  • --metadata accepts key=value pairs in addition to JSON by @baberabb in #4054
  • --use_cache keys are hashable for multimodal image/byte requests by @feiiiiii5 in #4040
  • Cache parent directories are created when missing by @sdivyanshu90 in #4047; delete_cache is a no-op when the directory is absent by @feiiiiii5 in #4035
  • A local directory sharing a task's name no longer shadows the registered task by @nloughl in #3670
  • Function resolving works for custom tasks by @SkyR0ver in #3992; string dictionary arguments parse correctly by @tandede in #4020
  • A scalar seed from a config file is normalized the same way the CLI does by @winklemad in #4003
  • Sample counts no longer depend on metric ordering by @arthi-arumugam-git in #4068
  • scripts/requests_caching.py entrypoint repaired — bad kwargs and a helper defined after its use by @Anai-Guo in #4059
  • chrF++ aggregation and metric (word_order=2) by @KrishVenky in #3780; TER metric direction corrected and the chrF docstring fixed by @borgr in #3993
  • All numpy scalar types serialize to JSON, not just int64/int32, by @iamsharduld in #3885

Migration Notes

  • Few-shot prompts changed for tasks using a document-specific gen_prefix (RULER niah_single_1, humaneval_instruct, humaneval_64_instruct) and anywhere the sampler previously drew the eval document into its own shots. Prior numbers on those tasks may not be comparable.
  • minerva_math and leaderboard_math_* now diverge. lm_eval/tasks/leaderboard/math/utils.py is deliberately frozen to reproduce Open LLM Leaderboard v2 scoring; the normalization fixes landed in minerva_math only.
  • global_piqa task names changed — see Task Changes above.
  • If you pinned datasets<4 to keep script-based tasks working, you can unpin. Out-of-tree task YAMLs pointing at script-based datasets still need the same treatment.
  • winml now sits on top of onnxruntime-genai and pulls in that extra.

<details>

<summary><h2>What's Changed</h2></summary>

  • feat: 0.4.13.dev0 by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3764
  • update global piqa by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3816
  • fix for keyerror while running humaneval_infilling by @jaydeepborkar in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768
  • fix: strip whitespace from input/target for FDA, SWDE, and SQuAD_completion tasks by @EphraiemSarabamoun in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795
  • Fix "trasnlation" typo in DarijaBench translation task names by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824
  • fix(ruler): replace defunct hotpot dataset host with pinned HF mirror by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3806
  • fix: keep Anthropic stop sequences nonempty by @he-yufeng in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822
  • fix(vllm): warn when device argument is ignored by @Apeironics in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803
  • fix sglang args by @wm901115nwpu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817
  • feat(legalbench): add HELM-lite LegalBench subset (5 tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860
  • Fix space in North Macedonian task identifiers (INCLUDE suite) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3848
  • fix(kormedmcqa): extract answer choice per paper Appendix B by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842
  • Fix dataset paths for xnli, xcopa, paws-x, and xquad by @Chessing234 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3870
  • Add IndicParam: MCQ benchmark for 12 low-resource Indic languages by @hjoshifonteva in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826
  • feat(portuguese_bench): add ASSIN2 RTE and STS tasks by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3812
  • fix(prost): load PROST without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977
  • fix(spanish_bench): load wnli_es without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3982
  • fix(tests): pass dataset_kwargs in test_download by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3980
  • fix(tasks): use namespaced dataset path for wsc273 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3946
  • fix(tasks): use namespaced dataset paths for webqs, medmcqa, and wmt16 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3942
  • fix(arithmetic): load arithmetic tasks without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3976
  • fix(mmlusr): restore dataset loading for all 171 MMLU-SR tasks by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3972
  • fix(scrolls): load SCROLLS without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3975
  • fix(tasks): use script-less parquet mirrors for mathqa, siqa, and moral_stories by @shubhangithub in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943
  • fix(tmmluplus): use ikala/tmmluplus so the task loads on datasets>=4 by @gowtham-sai-yadav in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950
  • Fix broken afrobench group task references (afrisenti/mafand prompt_2) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3847
  • fix(longbench): skip blank leading line in code_sim_score extraction by @cameronshinn in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921
  • fix(afrixnli): use Jinja braces in prompt_1 doc_to_text by @Solaris-star in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944
  • fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs by @hancheolcho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923
  • Fix fewshot_config.split precedence in TaskConfig by @chuenchen309 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937
  • Exclude eval docs from first-n few-shot samples by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978
  • fix: prevent ValueError when batch_size="auto" is passed to neuronx model by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971
  • fix: prevent ValueError when batch_size="auto:N" is passed to API models by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3970
  • fix: resolve fewshot gen_prefix against the fewshot doc, not the eval doc by @adityasingh2400 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979
  • Putnam Axiom by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3998
  • feat(legalbench): add Contract NLI suite (14 NDA entailment tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3954
  • Fix megatron_lm backend against Megatron-LM core_v0.18+: argument parsing moved out of initialize_megatron() by @oberpierre in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991
  • chore(ci): update pre-commit hooks and workflow packages by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4023
  • fix(hf): pass max_cpu_memory through to accelerate's max_memory by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4016
  • chore(megatron): run linter by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4024
  • fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal) by @OrionArchitekton in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916
  • Honor configured timeout for synchronous API requests by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3995
  • fix(api): support think_end_token for chat completions by @yaodong-shen in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959
  • feat(models): add cross-platform onnxruntime-genai backend + refactor winml by @thiagocrepaldi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960
  • Enable function resolving for custom tasks by @SkyR0ver in https://github.com/EleutherAI/lm-evaluation-harness/pull/3992
  • fix(config): parse dictionary strings from YAML configs by @tandede in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020
  • fix: normalize scalar seed from a config file the way the CLI does by @winklemad in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003
  • fix(filters): format_span only normalizes labels, not entity text by @k-dickinson in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887
  • fix: make delete_cache a no-op when the cache directory is absent by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035
  • Add Terretaqa task into Catalanbench by @ivanmartinezmurillo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784
  • Add make_table regression coverage by @aryanputta in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788
  • Add cieaCOVA task into catalan bench by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4043
  • fix(irokobench): repair afrimmlu/afrimgsm/afrixnli task registration and update READMEs by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3841
  • fix: fall back to tokenizer.eos_token when decode returns empty string by @ganeshr10 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657
  • fix(med_prescriptions): require both keys before combining complaints and diagnosis by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4041
  • fix: local directory with task name no longer shadows registered task by @nloughl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670
  • feat(models): add raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984
  • Align pubmedqa task name in README with yaml by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/4025
  • Serialize all numpy scalar types in JSON output, not just int64/int32 by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885
  • Fix TER metric direction (higher_is_better) and correct chrF docstring by @borgr in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993
  • test(registry): add tests for higher_is_better directions by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4050
  • Fix ContextSampler crash when few-shot pool contains duplicate eval_doc rows by @Kymi808 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790
  • Fix group stderr to match weight_by_size=False (unweighted) aggregation by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3882
  • Fix MultiChoiceRegexFilter prefix-shadowing of choice text by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3884
  • MMLU Task Name by @Teddygat0r in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660
  • Add mlqa tag to run all variants. by @ivanbaldo in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977
  • Create missing request-cache parent directories by @sdivyanshu90 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047
  • Fix the Unitxt init method to set the task name by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225
  • Pass dataset_kwargs for Unitxt tasks by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3230
  • feat: add chrf++ aggregation and metric (word_order=2) by @KrishVenky in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780
  • feat(tasks): add LongProc benchmark (6 task types, 16 configs) by @xiye17 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544
  • feat(tydiqa): add TyDiQA Gold Passage tasks (9 languages) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/4044
  • add GreekMMLU (official native-sourced benchmark) task configuration by @mersinkonomi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581
  • feat(cli): allow key=value to be passed to --metadata; linting by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4054
  • Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py by @vnayakde in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372
  • fix(minerva_math): correct few-shot prompt LaTeX by @Ben3892 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045
  • fix(minerva_math): score an answer identical to the gold as correct by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034
  • fix(minerva_math): thousands-separator comma strip fuses bare digit tuples ("0,1" -> "01") by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4039
  • fix(minerva_math): stop sqrt shorthand normalization from corrupting indexed roots by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4037
  • fix(ruler): resolve the tokenizer name before the cached lookup by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4048
  • chore(pre-commit) by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4060
  • fix(scripts): repair requests_caching.py entrypoint (bad kwargs + helper defined after use) by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4059
  • Add Uncheatable Eval by @ziqing-huang in https://github.com/EleutherAI/lm-evaluation-harness/pull/3442
  • Add IndicXNLI Gujarati task (indicxnli_gu) by @bhaumik611 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056
  • feat(physics_gre): add InflectionAI Physics GRE multiple-choice task by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3853
  • fix(evaluator): report a sample count that does not depend on metric order by @arthi-arumugam-git in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068
  • Fix/indicparam citation by @viveks-codes in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066
  • fix: make --use_cache keys hashable for multimodal image/byte requests by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4040

</details>

New Contributors

  • @jaydeepborkar made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768
  • @EphraiemSarabamoun made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795
  • @DaoyuanLi2816 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824
  • @he-yufeng made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822
  • @Apeironics made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803
  • @wm901115nwpu made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817
  • @bongho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860
  • @discobot made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842
  • @hjoshifonteva made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826
  • @ayaangazali made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977
  • @shubhangithub made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943
  • @gowtham-sai-yadav made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950
  • @cameronshinn made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921
  • @Solaris-star made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944
  • @hancheolcho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923
  • @chuenchen309 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937
  • @SiavashShams made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978
  • @AbdullahRasheed45 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971
  • @adityasingh2400 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979
  • @oberpierre made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991
  • @OrionArchitekton made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916
  • @yaodong-shen made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959
  • @thiagocrepaldi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960
  • @tandede made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020
  • @winklemad made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003
  • @k-dickinson made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887
  • @feiiiiii5 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035
  • @ivanmartinezmurillo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784
  • @aryanputta made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788
  • @ganeshr10 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657
  • @nloughl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670
  • @amd-sourjya made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984
  • @iamsharduld made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885
  • @borgr made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993
  • @Kymi808 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790
  • @Teddygat0r made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660
  • @ivanbaldo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977
  • @sdivyanshu90 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047
  • @mprahl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225
  • @KrishVenky made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780
  • @xiye17 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544
  • @mersinkonomi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581
  • @vnayakde made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372
  • @Ben3892 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045
  • @nata2627 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034
  • @bhaumik611 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056
  • @arthi-arumugam-git made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068
  • @viveks-codes made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066

Full Changelog: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.4.12...v0.4.13

What's Changed

  • feat: 0.4.13.dev0 by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3764
  • update global piqa by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3816
  • fix for keyerror while running humaneval_infilling by @jaydeepborkar in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768
  • fix: strip whitespace from input/target for FDA, SWDE, and SQuAD_completion tasks by @EphraiemSarabamoun in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795
  • Fix "trasnlation" typo in DarijaBench translation task names by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824
  • fix(ruler): replace defunct hotpot dataset host with pinned HF mirror by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3806
  • fix: keep Anthropic stop sequences nonempty by @he-yufeng in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822
  • fix(vllm): warn when device argument is ignored by @Apeironics in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803
  • fix sglang args by @wm901115nwpu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817
  • feat(legalbench): add HELM-lite LegalBench subset (5 tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860
  • Fix space in North Macedonian task identifiers (INCLUDE suite) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3848
  • fix(kormedmcqa): extract answer choice per paper Appendix B by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842
  • Fix dataset paths for xnli, xcopa, paws-x, and xquad by @Chessing234 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3870
  • Add IndicParam: MCQ benchmark for 12 low-resource Indic languages by @hjoshifonteva in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826
  • feat(portuguese_bench): add ASSIN2 RTE and STS tasks by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3812
  • fix(prost): load PROST without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977
  • fix(spanish_bench): load wnli_es without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3982
  • fix(tests): pass dataset_kwargs in test_download by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3980
  • fix(tasks): use namespaced dataset path for wsc273 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3946
  • fix(tasks): use namespaced dataset paths for webqs, medmcqa, and wmt16 by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3942
  • fix(arithmetic): load arithmetic tasks without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3976
  • fix(mmlusr): restore dataset loading for all 171 MMLU-SR tasks by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3972
  • fix(scrolls): load SCROLLS without the removed dataset script by @ayaangazali in https://github.com/EleutherAI/lm-evaluation-harness/pull/3975
  • fix(tasks): use script-less parquet mirrors for mathqa, siqa, and moral_stories by @shubhangithub in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943
  • fix(tmmluplus): use ikala/tmmluplus so the task loads on datasets>=4 by @gowtham-sai-yadav in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950
  • Fix broken afrobench group task references (afrisenti/mafand prompt_2) by @DaoyuanLi2816 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3847
  • fix(longbench): skip blank leading line in code_sim_score extraction by @cameronshinn in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921
  • fix(afrixnli): use Jinja braces in prompt_1 doc_to_text by @Solaris-star in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944
  • fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs by @hancheolcho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923
  • Fix fewshot_config.split precedence in TaskConfig by @chuenchen309 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937
  • Exclude eval docs from first-n few-shot samples by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978
  • fix: prevent ValueError when batch_size="auto" is passed to neuronx model by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971
  • fix: prevent ValueError when batch_size="auto:N" is passed to API models by @AbdullahRasheed45 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3970
  • fix: resolve fewshot gen_prefix against the fewshot doc, not the eval doc by @adityasingh2400 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979
  • Putnam Axiom by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3998
  • feat(legalbench): add Contract NLI suite (14 NDA entailment tasks) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3954
  • Fix megatron_lm backend against Megatron-LM core_v0.18+: argument parsing moved out of initialize_megatron() by @oberpierre in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991
  • chore(ci): update pre-commit hooks and workflow packages by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4023
  • fix(hf): pass max_cpu_memory through to accelerate's max_memory by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4016
  • chore(megatron): run linter by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4024
  • fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal) by @OrionArchitekton in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916
  • Honor configured timeout for synchronous API requests by @SiavashShams in https://github.com/EleutherAI/lm-evaluation-harness/pull/3995
  • fix(api): support think_end_token for chat completions by @yaodong-shen in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959
  • feat(models): add cross-platform onnxruntime-genai backend + refactor winml by @thiagocrepaldi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960
  • Enable function resolving for custom tasks by @SkyR0ver in https://github.com/EleutherAI/lm-evaluation-harness/pull/3992
  • fix(config): parse dictionary strings from YAML configs by @tandede in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020
  • fix: normalize scalar seed from a config file the way the CLI does by @winklemad in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003
  • fix(filters): format_span only normalizes labels, not entity text by @k-dickinson in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887
  • fix: make delete_cache a no-op when the cache directory is absent by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035
  • Add Terretaqa task into Catalanbench by @ivanmartinezmurillo in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784
  • Add make_table regression coverage by @aryanputta in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788
  • Add cieaCOVA task into catalan bench by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4043
  • fix(irokobench): repair afrimmlu/afrimgsm/afrixnli task registration and update READMEs by @discobot in https://github.com/EleutherAI/lm-evaluation-harness/pull/3841
  • fix: fall back to tokenizer.eos_token when decode returns empty string by @ganeshr10 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657
  • fix(med_prescriptions): require both keys before combining complaints and diagnosis by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4041
  • fix: local directory with task name no longer shadows registered task by @nloughl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670
  • feat(models): add raw onnxruntime backend for Model Builder ONNX exports by @amd-sourjya in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984
  • Align pubmedqa task name in README with yaml by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/4025
  • Serialize all numpy scalar types in JSON output, not just int64/int32 by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885
  • Fix TER metric direction (higher_is_better) and correct chrF docstring by @borgr in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993
  • test(registry): add tests for higher_is_better directions by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4050
  • Fix ContextSampler crash when few-shot pool contains duplicate eval_doc rows by @Kymi808 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790
  • Fix group stderr to match weight_by_size=False (unweighted) aggregation by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3882
  • Fix MultiChoiceRegexFilter prefix-shadowing of choice text by @iamsharduld in https://github.com/EleutherAI/lm-evaluation-harness/pull/3884
  • MMLU Task Name by @Teddygat0r in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660
  • Add mlqa tag to run all variants. by @ivanbaldo in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977
  • Create missing request-cache parent directories by @sdivyanshu90 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047
  • Fix the Unitxt init method to set the task name by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225
  • Pass dataset_kwargs for Unitxt tasks by @mprahl in https://github.com/EleutherAI/lm-evaluation-harness/pull/3230
  • feat: add chrf++ aggregation and metric (word_order=2) by @KrishVenky in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780
  • feat(tasks): add LongProc benchmark (6 task types, 16 configs) by @xiye17 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544
  • feat(tydiqa): add TyDiQA Gold Passage tasks (9 languages) by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/4044
  • add GreekMMLU (official native-sourced benchmark) task configuration by @mersinkonomi in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581
  • feat(cli): allow key=value to be passed to --metadata; linting by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4054
  • Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py by @vnayakde in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372
  • fix(minerva_math): correct few-shot prompt LaTeX by @Ben3892 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045
  • fix(minerva_math): score an answer identical to the gold as correct by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034
  • fix(minerva_math): thousands-separator comma strip fuses bare digit tuples ("0,1" -> "01") by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4039
  • fix(minerva_math): stop sqrt shorthand normalization from corrupting indexed roots by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4037
  • fix(ruler): resolve the tokenizer name before the cached lookup by @nata2627 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4048
  • chore(pre-commit) by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/4060
  • fix(scripts): repair requests_caching.py entrypoint (bad kwargs + helper defined after use) by @Anai-Guo in https://github.com/EleutherAI/lm-evaluation-harness/pull/4059
  • Add Uncheatable Eval by @ziqing-huang in https://github.com/EleutherAI/lm-evaluation-harness/pull/3442
  • Add IndicXNLI Gujarati task (indicxnli_gu) by @bhaumik611 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056
  • feat(physics_gre): add InflectionAI Physics GRE multiple-choice task by @bongho in https://github.com/EleutherAI/lm-evaluation-harness/pull/3853
  • fix(evaluator): report a sample count that does not depend on metric order by @arthi-arumugam-git in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068
  • Fix/indicparam citation by @viveks-codes in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066
  • fix: make --use_cache keys hashable for multimodal image/byte requests by @feiiiiii5 in https://github.com/EleutherAI/lm-evaluation-harness/pull/4040

New Contributors

  • @jaydeepborkar made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3768
  • @EphraiemSarabamoun made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3795
  • @DaoyuanLi2816 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3824
  • @he-yufeng made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3822
  • @Apeironics made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3803
  • @wm901115nwpu made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3817
  • @bongho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3860
  • @discobot made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3842
  • @hjoshifonteva made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3826
  • @ayaangazali made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3977
  • @shubhangithub made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3943
  • @gowtham-sai-yadav made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3950
  • @cameronshinn made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3921
  • @Solaris-star made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3944
  • @hancheolcho made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3923
  • @chuenchen309 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3937
  • @SiavashShams made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3978
  • @AbdullahRasheed45 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3971
  • @adityasingh2400 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3979
  • @oberpierre made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3991
  • @OrionArchitekton made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3916
  • @yaodong-shen made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3959
  • @thiagocrepaldi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3960
  • @tandede made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4020
  • @winklemad made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4003
  • @k-dickinson made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3887
  • @feiiiiii5 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4035
  • @ivanmartinezmurillo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3784
  • @aryanputta made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3788
  • @ganeshr10 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3657
  • @nloughl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3670
  • @amd-sourjya made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3984
  • @iamsharduld made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3885
  • @borgr made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3993
  • @Kymi808 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3790
  • @Teddygat0r made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3660
  • @ivanbaldo made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/2977
  • @sdivyanshu90 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4047
  • @mprahl made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3225
  • @KrishVenky made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3780
  • @xiye17 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3544
  • @mersinkonomi made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3581
  • @vnayakde made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3372
  • @Ben3892 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4045
  • @nata2627 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4034
  • @bhaumik611 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4056
  • @arthi-arumugam-git made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4068
  • @viveks-codes made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/4066

Full Changelog: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.4.12...v0.4.13

Files

EleutherAI/lm-evaluation-harness-v0.4.13.zip

Files (11.1 MB)

Name Size
md5:87de65d2dcbd9bd3aa54dd5837a00e8f
11.1 MB Preview Download

Additional details

Related works