Software Open Access
Wolf, Thomas; Debut, Lysandre; Sanh, Victor; Chaumond, Julien; Delangue, Clement; Moi, Anthony; Cistac, Perric; Ma, Clara; Jernite, Yacine; Plu, Julien; Xu, Canwen; Le Scao, Teven; Gugger, Sylvain; Drame, Mariama; Lhoest, Quentin; Rush, Alexander M.
v4.10.0: LayoutLM-v2, LayoutXLM, BEiT LayoutLM-v2 and LayoutXLM
Four new models are released as part of the LatourLM-v2 implementation:
LayoutLMv2ForQuestionAnswering, in PyTorch.
The LayoutLMV2 model was proposed in LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding by Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou. LayoutLMV2 improves LayoutLM to obtain state-of-the-art results across several document image understanding benchmarks:
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=layoutlmv2BEiT
Three new models are released as part of the BEiT implementation:
BeitForImageClassification, in PyTorch.
The BEiT model was proposed in BEiT: BERT Pre-Training of Image Transformers by Hangbo Bao, Li Dong and Furu Wei. Inspired by BERT, BEiT is the first paper that makes self-supervised pre-training of Vision Transformers (ViTs) outperform supervised pre-training. Rather than pre-training the model to predict the class of an image (as done in the original ViT paper), BEiT models are pre-trained to predict visual tokens from the codebook of OpenAI's DALL-E model given masked patches.
Compatible checkpoints can be found on the Hub: https://huggingface.co/models?filter=beitSpeech improvements
The Wav2Vec2 and HuBERT models now have a sequence classification head available.
The DeBERTa and DeBERTa-v2 models have been converted from PyTorch to TensorFlow.
EncoderDecoder, DistilBERT, and ALBERT, now have support in Flax!
A new example has been added in TensorFlow: multiple choice! Data collators have become framework agnostic and can now work for both TensorFlow and NumPy on top of PyTorch.
The Auto APIs have been disentangled from all the other mode modules of the Transformers library, so you can now safely import the Auto classes without importing all the models (and maybe getting errors if your setup is not compatible with one specific model). The actual model classes are only imported when needed.
When loading some kinds of corrupted state dictionaries of models, the
PreTrainedModel.from_pretrained method was sometimes silently ignoring weights. This has now become a real error.
classifier_dropoutto classification heads #12794 (@PhilipMay)
Seq2SeqTrainerset max_length and num_beams only when non None #12899 (@cchen-dialpad)
Trainer.evaluate()crash when using only tensorboardX #12963 (@aphedges)
Trainer.train(resume_from_checkpoint=False)is causing an exception #12981 (@PhilipMay)
inputs_embedsis used. #13128 (@sararb)
AutoModel.from_pretrained(..., torch_dtype=...)#13209 (@stas00)
model_types cannot be in the mapping #13259 (@LysandreJik)
tokenizer_class_from_namefor models with
-in the name #13251 (@stas00)
image-classificationpipeline to new testing #13272 (@Narsil)
summarizationpipeline to new testing format. #13279 (@Narsil)
table-question-answeringpipeline to new testing. #13280 (@Narsil)
table-question-answeringpipeline to new testing #13281 (@Narsil)
text2text-generationto new pipeline testing mecanism #13283 (@Narsil)
text-generationpipeline to new testing framework. #13285 (@Narsil)
token-classificationpipeline to new testing. #13286 (@Narsil)
translationpipeline to new testing scheme. #13297 (@Narsil)
zero-shot-classificationpipeline to new testing. #13299 (@Narsil)
return_tensorsargument too. #13301 (@Narsil)
[Flax] Correct all return tensors to numpy #13307 (@patrickvonplaten)
examples: only use keep_linebreaks when reading TXT files #13320 (@stefan-it)
dataloaderdoes not know the
batch_size. #13188 (@mbforbes)