Conference paper Open Access

Analysis of Positional Encodings for Neural Machine Translation

Rosendahl, Jan; Tran, Viet Anh Khoa; Wang, Weiyue; Ney, Hermann


MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00000nam##2200000uu#4500</leader>
  <controlfield tag="005">20200120173800.0</controlfield>
  <controlfield tag="001">3525024</controlfield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Human Language Technology and Pattern Recognition Group, RWTH Aachen University, Germany</subfield>
    <subfield code="a">Tran, Viet Anh Khoa</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Human Language Technology and Pattern Recognition Group, RWTH Aachen University, Germany</subfield>
    <subfield code="a">Wang, Weiyue</subfield>
  </datafield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">Human Language Technology and Pattern Recognition Group, RWTH Aachen University, Germany</subfield>
    <subfield code="a">Ney, Hermann</subfield>
  </datafield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">485651</subfield>
    <subfield code="z">md5:a2422d4b74a9634b2567ac4e9f630065</subfield>
    <subfield code="u">https://zenodo.org/record/3525024/files/IWSLT2019_paper_21.pdf</subfield>
  </datafield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2019-11-02</subfield>
  </datafield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">openaire</subfield>
    <subfield code="p">user-iwslt2019</subfield>
    <subfield code="o">oai:zenodo.org:3525024</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="u">Human Language Technology and Pattern Recognition Group, RWTH Aachen University, Germany</subfield>
    <subfield code="a">Rosendahl, Jan</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Analysis of Positional Encodings for Neural Machine Translation</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">user-iwslt2019</subfield>
  </datafield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="u">https://creativecommons.org/licenses/by/4.0/legalcode</subfield>
    <subfield code="a">Creative Commons Attribution 4.0 International</subfield>
  </datafield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2">opendefinition.org</subfield>
  </datafield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;In this work we analyze and compare the behavior of the Transformer architecture when using different positional encoding methods. While absolute and relative positional encoding perform equally strong overall, we show that relative positional encoding is vastly superior (4.4% to 11.9% BLEU) when translating a sentence that is longer than any observed training sentence. We further propose and analyze variations of relative positional encoding and observe that the number of trainable parameters can be reduced without a performance loss, by using fixed encoding vectors or by removing some of the positional encoding vectors.&lt;/p&gt;</subfield>
  </datafield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.3525023</subfield>
  </datafield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.3525024</subfield>
    <subfield code="2">doi</subfield>
  </datafield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">publication</subfield>
    <subfield code="b">conferencepaper</subfield>
  </datafield>
</record>
137
123
views
downloads
All versions This version
Views 137138
Downloads 123122
Data volume 59.7 MB59.2 MB
Unique views 126127
Unique downloads 117116

Share

Cite as