huggingface/datasets: 2.3.0

Datasets Changes

• New: ImageNet-Sketch by @nateraw in https://github.com/huggingface/datasets/pull/4301
• New: Biwi Kinect Head Pose by @dnaveenr in https://github.com/huggingface/datasets/pull/3903
• New: enwik8 by @HallerPatrick in https://github.com/huggingface/datasets/pull/4321
• New: LCCC dataset by @silverriver in https://github.com/huggingface/datasets/pull/4416
• New: TruthfulQA by @jon-tow in https://github.com/huggingface/datasets/pull/4159
• New: BIG-bench by @andersjohanandreassen in https://github.com/huggingface/datasets/pull/4125
• New: QuickDraw by @mariosasko in https://github.com/huggingface/datasets/pull/3592
• New: SST-2 by @albertvillanova in https://github.com/huggingface/datasets/pull/4473
• Update: imagenet-1k - remove manual download by @mariosasko in https://github.com/huggingface/datasets/pull/4299
  • ImageNet can now be loaded in python with load_dataset without requiring a manual download !
  • It also supports streaming mode with load_dataset("imagenet-1k", streaming=True)
• Update: spider - Remove Google Drive URL by @albertvillanova in https://github.com/huggingface/datasets/pull/4410
• Update: blended_skill_talk - add missing columns to by @mariosasko in https://github.com/huggingface/datasets/pull/4437
• Update: multi-news - Use newer version with fixes by @JohnGiorgi in https://github.com/huggingface/datasets/pull/4451
• Update: fever - update data URLs by @albertvillanova in https://github.com/huggingface/datasets/pull/44554459
• Update: udhr - Add and fix language tags by @albertvillanova in https://github.com/huggingface/datasets/pull/
• Update: udhr - update metadata by @leondz in https://github.com/huggingface/datasets/pull/4362
• Update: wider_face - Replace data URLs once hosted on the Hub by @albertvillanova in https://github.com/huggingface/datasets/pull/4469
• Update: PASS - update dataset version by @mariosasko in https://github.com/huggingface/datasets/pull/4488
• Fix: GEM - fix bug in wiki_auto_asset_turk config by @albertvillanova in https://github.com/huggingface/datasets/pull/4389
• Fix: GEM - fix URL for totto config by @albertvillanova in https://github.com/huggingface/datasets/pull/4396
• Fix: timit_asr - fix DuplicatedKeysError by @albertvillanova in https://github.com/huggingface/datasets/pull/4424
• Fix: timit_asr - Make extensions case-insensitive by @albertvillanova in https://github.com/huggingface/datasets/pull/4425
• Fix: timit_asr - Fix directory names for LDC data by @albertvillanova in https://github.com/huggingface/datasets/pull/4436
• Fix: iwslt2017 by @lhoestq in https://github.com/huggingface/datasets/pull/4481

Dataset Features

• to_tf_dataset rewrite by @Rocketknight1 in https://github.com/huggingface/datasets/pull/4170
  • see more in the documentation
• Support DataLoader with num_workers > 0 in streaming mode by @lhoestq in https://github.com/huggingface/datasets/pull/4375
  • see more in the documentation
• Added stratify option to train_test_split by @nandwalritik in https://github.com/huggingface/datasets/pull/4322
• Re-add support for Apache Beam functionality by @albertvillanova in https://github.com/huggingface/datasets/pull/4328
• Resume push_to_hub: skip identical files in push_to_hub instead of overwriting by @mariosasko in https://github.com/huggingface/datasets/pull/4402
• Support nested/complex feature types as features in packaged loaders by @mariosasko in https://github.com/huggingface/datasets/pull/4364

Dataset Cards

• Minor fixes/improvements in scene_parse_150 card by @mariosasko in https://github.com/huggingface/datasets/pull/4447
• Tidy up license metadata for google_wellformed_query, newspop, sick by @leondz in https://github.com/huggingface/datasets/pull/4378
• Fix example in opus_ubuntu, Add license info by @leondz in https://github.com/huggingface/datasets/pull/4360
• Update README.md of fquad by @lhoestq in https://github.com/huggingface/datasets/pull/4450

Documentation

• Add API code examples for loading methods by @stevhliu in https://github.com/huggingface/datasets/pull/4300
• Add API code examples for remaining main classes by @stevhliu in https://github.com/huggingface/datasets/pull/4292
• Generalize tutorials for audio and vision by @stevhliu in https://github.com/huggingface/datasets/pull/4468

Other improvements and bug fixes

• Update CI deprecated legacy image by @albertvillanova in https://github.com/huggingface/datasets/pull/4393
• remove int documentation from logging docs by @lvwerra in https://github.com/huggingface/datasets/pull/4392
• Fix docstring in DatasetDict::shuffle by @felixdivo in https://github.com/huggingface/datasets/pull/4344
• Fix Version equality by @albertvillanova in https://github.com/huggingface/datasets/pull/4359
• Set builder name from module instead of class by @albertvillanova in https://github.com/huggingface/datasets/pull/4388
• Test dill by @albertvillanova in https://github.com/huggingface/datasets/pull/4385
• Refactor download by @albertvillanova in https://github.com/huggingface/datasets/pull/4384
• Fix dependency on dill version by @albertvillanova in https://github.com/huggingface/datasets/pull/4397
• Support remote cache_dir by @albertvillanova in https://github.com/huggingface/datasets/pull/4347
• Update imagenet gate by @lhoestq in https://github.com/huggingface/datasets/pull/4408
• Fix dataset builder default version by @albertvillanova in https://github.com/huggingface/datasets/pull/4356
• Uncomment logging deactivation for ArrowBasedBuilder by @thomasw21 in https://github.com/huggingface/datasets/pull/4403
• Rename DatasetBuilder config_name by @albertvillanova in https://github.com/huggingface/datasets/pull/4414
• Fix metadata validation by @albertvillanova in https://github.com/huggingface/datasets/pull/4390
• Add HF.co for PRs/Issues for specific datasets by @lhoestq in https://github.com/huggingface/datasets/pull/4427
• Fix type hint and documentation for new_fingerprint by @fxmarty in https://github.com/huggingface/datasets/pull/4326
• Skip hidden files/directories in data files resolution and iter_files by @mariosasko in https://github.com/huggingface/datasets/pull/4412
• Fix docstring of inspect_dataset by @albertvillanova in https://github.com/huggingface/datasets/pull/4438
• Fix builder docstring by @albertvillanova in https://github.com/huggingface/datasets/pull/4432
• Fix kwargs in docstrings by @albertvillanova in https://github.com/huggingface/datasets/pull/4444
• Fix missing args in docstring of load_dataset_builder by @albertvillanova in https://github.com/huggingface/datasets/pull/4445
• Add missing kwargs to docstrings by @albertvillanova in https://github.com/huggingface/datasets/pull/4446
• Add extractor for bzip2-compressed files by @asivokon in https://github.com/huggingface/datasets/pull/4421
• Fix dummy dataset generation script for handling nested types of _URLs by @silverriver in https://github.com/huggingface/datasets/pull/4434
• Update dataset_infos.json with new split info in Dataset.push_to_hub to avoid verification error by @mariosasko in https://github.com/huggingface/datasets/pull/4415
• Update builder docstring for deprecated/added arguments by @albertvillanova in https://github.com/huggingface/datasets/pull/4429
• Extend support for streaming datasets that use xml.dom.minidom.parse by @albertvillanova in https://github.com/huggingface/datasets/pull/4464
• Fix script fetching and local path handling in inspect_dataset and inspect_metric by @mariosasko in https://github.com/huggingface/datasets/pull/4433
• Fix bigbench config names by @lhoestq in https://github.com/huggingface/datasets/pull/4465
• Fix 401 error for unauthticated requests to non-existing repos by @lhoestq in https://github.com/huggingface/datasets/pull/4472
• Reorder returned validation/test splits in script template by @albertvillanova in https://github.com/huggingface/datasets/pull/4470
• Better ImportError message when a dataset script dependency is missing by @lhoestq in https://github.com/huggingface/datasets/pull/4484
• Fix cast to null by @lhoestq in https://github.com/huggingface/datasets/pull/4485
• [Docs] How to use with PyTorch page by @lhoestq in https://github.com/huggingface/datasets/pull/4474
• Optimize contiguous shard and select by @lhoestq in https://github.com/huggingface/datasets/pull/4466
• First draft of the docs for TF + Datasets by @Rocketknight1 in https://github.com/huggingface/datasets/pull/4457
• Update _format_columns in remove_columns by @alvarobartt in https://github.com/huggingface/datasets/pull/4411
• Fix wrong map parameter name in cache docs by @h4iku in https://github.com/huggingface/datasets/pull/4293
• Pin the revision in imagenet download links by @lhoestq in https://github.com/huggingface/datasets/pull/4492
• Refactor column mappings for question answering datasets by @lewtun in https://github.com/huggingface/datasets/pull/4391

New Contributors

• @leondz made their first contribution in https://github.com/huggingface/datasets/pull/4378
• @felixdivo made their first contribution in https://github.com/huggingface/datasets/pull/4344
• @nandwalritik made their first contribution in https://github.com/huggingface/datasets/pull/4322
• @fxmarty made their first contribution in https://github.com/huggingface/datasets/pull/4326
• @HallerPatrick made their first contribution in https://github.com/huggingface/datasets/pull/4321
• @silverriver made their first contribution in https://github.com/huggingface/datasets/pull/4416
• @asivokon made their first contribution in https://github.com/huggingface/datasets/pull/4421
• @andersjohanandreassen made their first contribution in https://github.com/huggingface/datasets/pull/4125

Full Changelog: https://github.com/huggingface/datasets/compare/2.2.2...lol