Transformers: State-of-the-Art Natural Language Processing

Perceiver

The Perceiver model was released in the previous version:

Perceiver

Eight new models are released as part of the Perceiver implementation: PerceiverModel, PerceiverForMaskedLM, PerceiverForSequenceClassification, PerceiverForImageClassificationLearned, PerceiverForImageClassificationFourier, PerceiverForImageClassificationConvProcessing, PerceiverForOpticalFlow, PerceiverForMultimodalAutoencoding, in PyTorch.

The Perceiver IO model was proposed in Perceiver IO: A General Architecture for Structured Inputs & Outputs by Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, João Carreira.

• Add Perceiver IO by @NielsRogge in https://github.com/huggingface/transformers/pull/14487

Compatible checkpoints can be found on the hub: https://huggingface.co/models?other=perceiver

Version v4.14.0 adds support for Perceiver in multiple pipelines, including the fill mask and sequence classification pipelines.

The Keras push to hub callback now generates model cards when pushing to the model hub. Additionally to the callback, model cards will be generated by default by the model.push_to_hub() method.

• TF model cards by @Rocketknight1 in https://github.com/huggingface/transformers/pull/14720

• Fix : wrong link in the documentation (ConvBERT vs DistilBERT) by @Tikquuss in https://github.com/huggingface/transformers/pull/14705
• Put back open in colab markers by @sgugger in https://github.com/huggingface/transformers/pull/14684
• Fix doc examples: KeyError by @ydshieh in https://github.com/huggingface/transformers/pull/14699
• Fix doc examples: 'CausalLMOutput...' object has no attribute 'last_hidden_state' by @ydshieh in https://github.com/huggingface/transformers/pull/14678
• Adding Perceiver to AutoTokenizer. by @Narsil in https://github.com/huggingface/transformers/pull/14711
• Fix doc examples: unexpected keyword argument by @ydshieh in https://github.com/huggingface/transformers/pull/14689
• Automatically build doc notebooks by @sgugger in https://github.com/huggingface/transformers/pull/14718
• Fix special character in MDX by @sgugger in https://github.com/huggingface/transformers/pull/14721
• Fixing tests for perceiver (texts) by @Narsil in https://github.com/huggingface/transformers/pull/14719
• [doc] document MoE model approach and current solutions by @stas00 in https://github.com/huggingface/transformers/pull/14725
• [Flax examples] remove dependancy on pytorch training args by @patil-suraj in https://github.com/huggingface/transformers/pull/14636
• Update bug-report.md by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14715
• [Adafactor] Fix adafactor by @patrickvonplaten in https://github.com/huggingface/transformers/pull/14713
• Code parrot minor fixes/niceties by @ncoop57 in https://github.com/huggingface/transformers/pull/14666
• Fix doc examples: modify config before super().init by @ydshieh in https://github.com/huggingface/transformers/pull/14697
• Improve documentation of some models by @NielsRogge in https://github.com/huggingface/transformers/pull/14695
• Skip Perceiver tests by @LysandreJik in https://github.com/huggingface/transformers/pull/14745
• Add ability to get a list of supported pipeline tasks by @codesue in https://github.com/huggingface/transformers/pull/14732
• Fix the perceiver docs by @LysandreJik in https://github.com/huggingface/transformers/pull/14748
• [CI/pt-nightly] switch to cuda-11.3 by @stas00 in https://github.com/huggingface/transformers/pull/14726
• Swap TF and PT code inside two blocks by @LucienShui in https://github.com/huggingface/transformers/pull/14742
• Fix doc examples: cannot import name by @ydshieh in https://github.com/huggingface/transformers/pull/14698
• Fix: change tooslow to slow by @ydshieh in https://github.com/huggingface/transformers/pull/14734
• Small fixes for the doc by @sgugger in https://github.com/huggingface/transformers/pull/14751
• Update transformers metadata by @sgugger in https://github.com/huggingface/transformers/pull/14724
• Mention no images added to repository by @LysandreJik in https://github.com/huggingface/transformers/pull/14738
• Avoid using tf.tile in embeddings for TF models by @ydshieh in https://github.com/huggingface/transformers/pull/14735
• Change how to load config of XLNetLMHeadModel by @josutk in https://github.com/huggingface/transformers/pull/14746
• Improve perceiver by @NielsRogge in https://github.com/huggingface/transformers/pull/14750
• Convert Trainer doc page to MarkDown by @sgugger in https://github.com/huggingface/transformers/pull/14753
• Update Table of Contents by @sgugger in https://github.com/huggingface/transformers/pull/14755
• Fixing tests for Perceiver by @Narsil in https://github.com/huggingface/transformers/pull/14739
• Make data shuffling in run_clm_flax.py respect global seed by @bminixhofer in https://github.com/huggingface/transformers/pull/13410
• Adding support for multiple mask tokens. by @Narsil in https://github.com/huggingface/transformers/pull/14716
• Fix broken links to distillation on index page of documentation by @amitness in https://github.com/huggingface/transformers/pull/14722
• [doc] performance: groups of operations by compute-intensity by @stas00 in https://github.com/huggingface/transformers/pull/14757
• Fix the doc_build_test job by @sgugger in https://github.com/huggingface/transformers/pull/14774
• Fix preprocess_function in run_summarization_flax.py by @ydshieh in https://github.com/huggingface/transformers/pull/14769

• Simplify T5 docs by @xhlulu in https://github.com/huggingface/transformers/pull/14776

• Update Perceiver code examples by @NielsRogge in https://github.com/huggingface/transformers/pull/14783

• @Tikquuss made their first contribution in https://github.com/huggingface/transformers/pull/14705
• @codesue made their first contribution in https://github.com/huggingface/transformers/pull/14732
• @LucienShui made their first contribution in https://github.com/huggingface/transformers/pull/14742
• @josutk made their first contribution in https://github.com/huggingface/transformers/pull/14746
• @amitness made their first contribution in https://github.com/huggingface/transformers/pull/14722

Full Changelog: https://github.com/huggingface/transformers/compare/v4.13.0...v4.14.0