Software Open Access

Transformers: State-of-the-Art Natural Language Processing

Wolf, Thomas; Debut, Lysandre; Sanh, Victor; Chaumond, Julien; Delangue, Clement; Moi, Anthony; Cistac, Perric; Ma, Clara; Jernite, Yacine; Plu, Julien; Xu, Canwen; Le Scao, Teven; Gugger, Sylvain; Drame, Mariama; Lhoest, Quentin; Rush, Alexander M.


Dublin Core Export

<?xml version='1.0' encoding='utf-8'?>
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
  <dc:creator>Wolf, Thomas</dc:creator>
  <dc:creator>Debut, Lysandre</dc:creator>
  <dc:creator>Sanh, Victor</dc:creator>
  <dc:creator>Chaumond, Julien</dc:creator>
  <dc:creator>Delangue, Clement</dc:creator>
  <dc:creator>Moi, Anthony</dc:creator>
  <dc:creator>Cistac, Perric</dc:creator>
  <dc:creator>Ma, Clara</dc:creator>
  <dc:creator>Jernite, Yacine</dc:creator>
  <dc:creator>Plu, Julien</dc:creator>
  <dc:creator>Xu, Canwen</dc:creator>
  <dc:creator>Le Scao, Teven</dc:creator>
  <dc:creator>Gugger, Sylvain</dc:creator>
  <dc:creator>Drame, Mariama</dc:creator>
  <dc:creator>Lhoest, Quentin</dc:creator>
  <dc:creator>Rush, Alexander M.</dc:creator>
  <dc:date>2020-10-01</dc:date>
  <dc:description>New Model additions
WavLM
WavLM was proposed in WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Furu Wei.
WavLM sets a new SOTA on the SUPERB benchmark.
Compatible checkpoints can be found on the hub: https://huggingface.co/models?other=wavlm

Add WavLM by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14354

Wav2Vec2Phoneme
Wav2Vec2Phoneme was proposed in Simple and Effective Zero-shot Cross-lingual Phoneme Recognition by Qiantong Xu, Alexei Baevski, Michael Auli.
Wav2Vec2Phoneme allows to do phoneme classification as part of automatic speech recognition

[Wav2Vec2 Phoneme] Let phonemizer lang default to tokenizer's settings by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14829

Compatible checkpoints can be found on the hub: https://huggingface.co/models?other=phoneme-recognition
UniSpeech-SAT
Unispeech-SAT was proposed in UNISPEECH-SAT: UNIVERSAL SPEECH REPRESENTATION LEARNING WITH SPEAKER AWARE PRE-TRAINING by Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu.
UniSpeech-SAT is especially good at speaker related tasks.
Compatible checkpoints can be found on the hub: https://huggingface.co/models?other=unispeech-sat
UniSpeech
Unispeech was proposed in UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data by Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang.
Three new models are released as part of the ImageGPT integration: ImageGPTModel, ImageGPTForCausalImageModeling, ImageGPTForImageClassification, in PyTorch.
Compatible checkpoints can be found on the hub: https://huggingface.co/models?other=unispeech
New Tasks
Speaker Diarization and Verification
Wav2Vec2-like architecture now have a speaker diarization and speaker verification head added to their architectures. 
You can try out the new task here: https://huggingface.co/spaces/microsoft/wavlm-speaker-verification

Add Speaker Diarization and Verification heads by @anton-l in https://github.com/huggingface/transformers/pull/14723

What's Changed

Move import to avoid circular import by @sgugger in https://github.com/huggingface/transformers/pull/14787
PoC for conserving old links by @sgugger in https://github.com/huggingface/transformers/pull/14754
Removes images to put them in a dataset by @LysandreJik in https://github.com/huggingface/transformers/pull/14781
Post sphinx-clean up and contributing guide updates by @sgugger in https://github.com/huggingface/transformers/pull/14790
Fix the build documentation job by @sgugger in https://github.com/huggingface/transformers/pull/14788
Update CONTRIBUTING.md by @kamalkraj in https://github.com/huggingface/transformers/pull/14799
Update CONTRIBUTING.md by @kamalkraj in https://github.com/huggingface/transformers/pull/14800
Train step fix by @Rocketknight1 in https://github.com/huggingface/transformers/pull/14796
[Generate] Make generate multi-modal by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14784
Remove require_datasets testing utility  by @LysandreJik in https://github.com/huggingface/transformers/pull/14795
[WavLM] Correct position bias computation by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14805
Fix Perceiver multi GPU test by @NielsRogge in https://github.com/huggingface/transformers/pull/14810
[WavLM] Layerdrop is not allowed for first layer by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14811
[Generate] Correct input_ids detection by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14815
Implement head_mask for Flax BERT and other models copied from BERT by @stancld in https://github.com/huggingface/transformers/pull/14620
Convert rst to mdx bert by @LysandreJik in https://github.com/huggingface/transformers/pull/14806
Wav2Vec2 meets phonemes by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14353
[ImageGPT] Deprecate pixel_values input name to input_ids by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14801
[Seq2SeqTrainer] Remove model input name hack by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14802
[WavLM] Fix slow tests by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14845
Add SD and SV heads for WavLM by @anton-l in https://github.com/huggingface/transformers/pull/14847
Add an argument to set bucket_cap_mb for PyTorch DDP by @changlan in https://github.com/huggingface/transformers/pull/14756
Update CONTRIBUTING.md by @kamalkraj in https://github.com/huggingface/transformers/pull/14835
Fix dead link to benchmarks.ipynb by @DerekChia in https://github.com/huggingface/transformers/pull/14842
[Perceiver] Skip multi-gpu tests for now by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14813
Add 'with torch.no_grad()' to DeBERTa integration test forward pass by @henholm in https://github.com/huggingface/transformers/pull/14821
Add 'with torch.no_grad()' to BERT integration test forward pass by @henholm in https://github.com/huggingface/transformers/pull/14820
Add a main_input_name attribute to all models by @sgugger in https://github.com/huggingface/transformers/pull/14803
[doc] typo by @stas00 in https://github.com/huggingface/transformers/pull/14849
[logging] implement warning_advice / TRANSFORMERS_NO_ADVISORY_WARNINGS by @stas00 in https://github.com/huggingface/transformers/pull/14669
Make the onnx submodule init lazy by @sgugger in https://github.com/huggingface/transformers/pull/14855
Convert docstrings of modeling files by @sgugger in https://github.com/huggingface/transformers/pull/14850
[Bart] better error message by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14854
Only create the model card on process 0 by @sgugger in https://github.com/huggingface/transformers/pull/14857
[ASR example] Improve example + add more examples by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14848
Fix the value error typo of AdamW's betas' valid values checking by @dourgey in https://github.com/huggingface/transformers/pull/14780
Add custom stopping_criteria and  logits_processor to generate by @lvwerra in https://github.com/huggingface/transformers/pull/14779
Replace commit sha by commit url for update jobs by @sgugger in https://github.com/huggingface/transformers/pull/14852
[examples/summarization] deal with None in data records by @stas00 in https://github.com/huggingface/transformers/pull/14816
[doc porting] several docs by @stas00 in https://github.com/huggingface/transformers/pull/14858
Mass conversion of documentation from rst to Markdown by @sgugger in https://github.com/huggingface/transformers/pull/14866
Fix FLAX_MULTIPLE_CHOICE_SAMPLE typo by @mishig25 in https://github.com/huggingface/transformers/pull/14871
Fixes in marian doc by @sgugger in https://github.com/huggingface/transformers/pull/14872
Fix FlaxMarianMTModel return block. by @sgugger in https://github.com/huggingface/transformers/pull/14873
Fix doc mistakes by @sgugger in https://github.com/huggingface/transformers/pull/14874
Convert model files from rst to mdx by @LysandreJik in https://github.com/huggingface/transformers/pull/14865
update the arguments add_prefix_space and trim_offsets in backend_tokenizer.post_processor of RobertaTokenizerFast by @SaulLu in https://github.com/huggingface/transformers/pull/14752
Feature/fix slow test in mluke by @Ryou0634 in https://github.com/huggingface/transformers/pull/14749
Updated deberta attention by @guillaume-be in https://github.com/huggingface/transformers/pull/14625
IterableDatasetShard should use per device batch size instead of real… by @SysuCharon in https://github.com/huggingface/transformers/pull/14714
Fix Perceiver code example by @NielsRogge in https://github.com/huggingface/transformers/pull/14879
Fix pytorch image classification example by @mariosasko in https://github.com/huggingface/transformers/pull/14883
Onnx enable tasks for supported models (part 2) by @michaelbenayoun in https://github.com/huggingface/transformers/pull/14700
Properly indent return block by @sgugger in https://github.com/huggingface/transformers/pull/14887

New Contributors

@changlan made their first contribution in https://github.com/huggingface/transformers/pull/14756
@DerekChia made their first contribution in https://github.com/huggingface/transformers/pull/14842
@henholm made their first contribution in https://github.com/huggingface/transformers/pull/14821
@dourgey made their first contribution in https://github.com/huggingface/transformers/pull/14780
@SysuCharon made their first contribution in https://github.com/huggingface/transformers/pull/14714

Full Changelog: https://github.com/huggingface/transformers/compare/v4.14.0...v4.15.0</dc:description>
  <dc:description>If you use this software, please cite it using these metadata.</dc:description>
  <dc:identifier>https://zenodo.org/record/5799695</dc:identifier>
  <dc:identifier>10.5281/zenodo.5799695</dc:identifier>
  <dc:identifier>oai:zenodo.org:5799695</dc:identifier>
  <dc:relation>url:https://github.com/huggingface/transformers/tree/v4.15.0</dc:relation>
  <dc:relation>doi:10.5281/zenodo.3385997</dc:relation>
  <dc:relation>url:https://zenodo.org/communities/zenodo</dc:relation>
  <dc:rights>info:eu-repo/semantics/openAccess</dc:rights>
  <dc:title>Transformers: State-of-the-Art Natural Language Processing</dc:title>
  <dc:type>info:eu-repo/semantics/other</dc:type>
  <dc:type>software</dc:type>
</oai_dc:dc>
37,139
1,293
views
downloads
All versions This version
Views 37,139942
Downloads 1,29325
Data volume 10.0 GB241.7 MB
Unique views 30,889869
Unique downloads 66724

Share

Cite as