Published December 8, 2022
| Version v0.3.0
Software
Open
EleutherAI/lm-evaluation-harness: v0.3.0
Authors/Creators
- Leo Gao
- Jonathan Tow
- Stella Biderman1
- Charles Lovering
- Jason Phang
- Anish Thite2
- Fazz
- Niklas Muennighoff3
- Thomas Wang4
- sdtblck
- tttyuntian
- researcher2
- Zdeněk Kasner5
- Khalid Almubarak
- Jeffrey Hsu6
- Pawan Sasanka Ammanamanchi
- Dirk Groeneveld
- Eric Tang
- Charles Foster
- kkawamu1
- xagi-dev
- uyhcire
- Andy Zou
- Ben Wang
- Jordan Clive7
- igor0
- Kevin Wang
- Nicholas Kross
- Fabrizio Milo8
- silentv0x
- 1. Booz Allen Hamilton
- 2. @ClarosAI
- 3. PKU
- 4. Hugging Face
- 5. @ufal
- 6. Ivy Natal
- 7. Chattermill
- 8. @recurrency
Description
HuggingFace Datasets Integration
This release integrates HuggingFace datasets as the core dataset management interface, removing previous custom downloaders.
- Refactor
Taskdownloading to useHuggingFace.datasetsby @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/300 - Add templates and update docs by @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/308
- Add dataset features to
TriviaQAby @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/305 - Add
SWAGby @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/306 - Fixes for using lm_eval as a library by @dirkgr in https://github.com/EleutherAI/lm-evaluation-harness/pull/309
- Researcher2 by @researcher2 in https://github.com/EleutherAI/lm-evaluation-harness/pull/261
- Suggested updates for the task guide by @StephenHogg in https://github.com/EleutherAI/lm-evaluation-harness/pull/301
- Add pre-commit by @Mistobaan in https://github.com/EleutherAI/lm-evaluation-harness/pull/317
- Decontam import fix by @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/321
- Add bootstrap_iters kwarg by @Muennighoff in https://github.com/EleutherAI/lm-evaluation-harness/pull/322
- Update decontamination.md by @researcher2 in https://github.com/EleutherAI/lm-evaluation-harness/pull/331
- Fix key access in squad evaluation metrics by @konstantinschulz in https://github.com/EleutherAI/lm-evaluation-harness/pull/333
- Fix make_disjoint_window for tail case by @richhankins in https://github.com/EleutherAI/lm-evaluation-harness/pull/336
- Manually concat tokenizer revision with subfolder by @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/343
- [deps] Use minimum versioning for
numexprby @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/352 - Remove custom datasets that are in HF by @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/330
- Add
TextSynthAPI by @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/299 - Add the original
LAMBADAdataset by @jon-tow in https://github.com/EleutherAI/lm-evaluation-harness/pull/357
- @dirkgr made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/309
- @Mistobaan made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/317
- @konstantinschulz made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/333
- @richhankins made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/336
Full Changelog: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.2.0...v0.3.0
Files
EleutherAI/lm-evaluation-harness-v0.3.0.zip
Files
(838.1 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:fe6d62ea5a5627c285414f0a4cff677d
|
838.1 kB | Preview Download |
Additional details
Related works
- Is supplement to
- https://github.com/EleutherAI/lm-evaluation-harness/tree/v0.3.0 (URL)