There is a newer version of the record available.

Published September 26, 2019 | Version v2.0.0
Software Open

huggingface/transformers: v2.0.0 - TF 2.0/PyTorch interoperability, improved tokenizers, improved torchscript support

Creators

  • 1. @huggingface
  • 2. Hugging Face
  • 3. DisAItek & Intel AI Innovators
  • 4. Stanford University
  • 5. @ShannonAI
  • 6. deepset
  • 7. Red Hat
  • 8. @ntu-spml-lab @Yoctol
  • 9. @JetBrains
  • 10. @allenai
  • 11. @RetroRabbit
  • 12. i2x
  • 13. @microsoft

Description

Name change: welcome 🤗 Transformers

Following the extension to TensorFlow 2.0, pytorch-transformers => transformers

Install with pip install transformers

TensorFlow 2.0 - PyTorch

All the PyTorch nn.Module classes now have their counterpart in TensorFlow 2.0 as tf.keras.Model classes. TensorFlow 2.0 classes have the same name as their PyTorch counterparts prefixed with TF.

The interoperability between TensorFlow and PyTorch is actually a lot deeper than what is usually meant when talking about libraries with multiple backends:

  • each model (not just the static computation graph) can be seamlessly moved from one framework to the other during the lifetime of the model for training/evaluation/usage (from_pretrained can load weights saved from models saved in one or the other framework),
  • an example is given in the quick-tour on TF 2.0 and PyTorch in the readme in which a model is trained using keras.fit before being opened in PyTorch for quick debugging/inspection.
Remaining unsupported operations in TF 2.0 (to be added later):
  • resizing input embeddings to add new tokens
  • pruning model heads
TPU support

Training on TPU using free TPUs provided in the TensorFlow Research Cloud (TFRC) program is possible but requires to implement a custom training loop (not possible with keras.fit at the moment). We will add an example of such a custom training loop soon.

Improved tokenizers

Tokenizers have been improved to provide extended encoding methods encoding_plus and additional arguments to encoding. Please refer to the doc for detailed usage of the new options.

Potential breaking change: positional order of some model keywords inputs changed

To be able to use Torchscript (see #1010, #1204 and #1195) the specific order of some models keywords inputs (attention_mask, token_type_ids...) has been changed.

If you used to call the models with keyword names for keyword arguments, e.g. model(inputs_ids, attention_mask=attention_mask, token_type_ids=token_type_ids), this should not cause any breaking change.

If you used to call the models with positional inputs for keyword arguments, e.g. model(inputs_ids, attention_mask, token_type_ids), you should double-check the exact order of input arguments.

Community additions/bug-fixes/improvements
  • new German model (@Timoeller)
  • new script for MultipleChoice training (SWAG, RocStories...) (@erenup)
  • better fp16 support (@ziliwang and @bryant1410)
  • fix evaluation in run_lm_finetuning (@SKRohit)
  • fiw LM finetuning to prevent crashing on assert len(tokens_b)>=1 (@searchivarius)
  • Various doc and docstring fixes (@sshleifer, @Maxpa1n, @mattolson93, @t080)

Files

huggingface/transformers-v2.0.0.zip

Files (937.2 kB)

Name Size Download all
md5:0796fe053523402a47419bf49cef910c
937.2 kB Preview Download

Additional details

Related works

Is supplement to
https://github.com/huggingface/transformers/tree/v2.0.0 (URL)