November 2, 2019 Conference paper Open Access

Nguyen, Toan Q.; Salazar, Julian

Citation Style Language JSON Export

{
  "publisher": "Zenodo", 
  "DOI": "10.5281/zenodo.3525484", 
  "language": "eng", 
  "title": "Transformers without Tears: Improving the Normalization of Self-Attention", 
  "issued": {
    "date-parts": [
      [
        2019, 
        11, 
        2
      ]
    ]
  }, 
  "abstract": "<p>We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PRENORM) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose&nbsp;l2&nbsp;normalization with a single scale parameter (SCALENORM) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FIXNORM). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT &#39;15 English-Vietnamese. We ob- serve sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT &#39;14 English-German), SCALENORM&nbsp;and FIXNORM&nbsp;remain competitive but PRENORM&nbsp;degrades performance.</p>", 
  "author": [
    {
      "family": "Nguyen, Toan Q."
    }, 
    {
      "family": "Salazar, Julian"
    }
  ], 
  "type": "paper-conference", 
  "id": "3525484"
}