Attention Is All You Need: The Transformer Paper's Five Consequential Claims
The 2017 Google Brain paper that scrapped recurrence and convolutions entirely, proposing the Transformer architecture that now underlies nearly every large language model.
We propose the Transformer, the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
In context
…prehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 27, 28, 22]. End-to-end memory networks are based on a recurrent attention mechanism instead of sequence- aligned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks [34]. To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence- aligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models…
We propose the Transformer, is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence- aligned sequence-aligned RNNs or convolution.
Receipt
- Source
- PDF · arxiv.org
- Read from
- Document text
- Located
- Page 2
- Speaker
- Not established
- Checked
- Sep 27, 2026
The Transformer (big) achieves 28.4 BLEU on WMT 2014 English-to-German, outperforming all previously reported models including ensembles by more than 2 BLEU, trained in 3.5 days on 8 P100 GPUs.
In context
…to be more unsure, but improves accuracy and BLEU score. 6 Results 6.1 Machine Translation On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2 . 0 BLEU, establishing a new state-of-the-art BLEU score of 28 . 4 . The configuration of this model is listed in the bottom line of Table 3. Training took 3 . 5 days on 8 P100 GPUs. Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models. On the WMT 2014 English-to-French translation task, our big model achieves a BLEU scor…
2014 English-to-German translation task, The big Transformer model (Transformer (big) in Table 2) outperforms the best achieves 28.4 BLEU on WMT 2014 English-to-German, outperforming all previously reported models including ensembles by more than 2 . 0 BLEU, trained in 3.5 days on 8 P100 GPUs.
Receipt
- Source
- PDF · arxiv.org
- Read from
- Document text
- Located
- Page 8
- Speaker
- Not established
- Checked
- Sep 27, 2026
A self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.
In context
…equential operations required. The third is the path length between long-range dependencies in the network. Learning long-range dependencies is a key challenge in many sequence transduction tasks. One key factor affecting the ability to learn such dependencies is the length of the paths forward and backward signals have to traverse in the network. The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare the maximum path length between any two input and output positions in networks composed of the different layer types. As noted in Table 1, a self-attention layer connects all p…
positions in networks composed of the different layer types. As noted in Table 1, A self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.
Receipt
- Source
- PDF · arxiv.org
- Read from
- Document text
- Located
- Page 6
- Speaker
- Not established
- Checked
- Sep 27, 2026
Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
In context
output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this. MultiHead( Q, K, V ) = Concat(head 1 , ..., head h ) W O where head i = Attention( QW Q i , KW K i , V W V i ) Where the projections are parameter matrices W Q i ∈ R d model × d k , W K i ∈ R d model × d k , W V i ∈ R d model × d v and W O ∈ R hd v × d model .
Receipt
- Source
- PDF · arxiv.org
- Read from
- Document text
- Located
- Page 5
- Speaker
- Not established
- Checked
- Sep 27, 2026
Label smoothing hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
In context
For the base model, we use a rate of P drop = 0 . 1 . Label Smoothing During training, we employed label smoothing of value ϵ ls = 0 . 1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score. 6 Results 6.1 Machine Translation On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2 . 0 BLEU, establishing a new state-of-the-art BLEU score of 28 .
Label smoothing of value ϵ ls = 0 . 1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
Receipt
- Source
- PDF · arxiv.org
- Read from
- Document text
- Located
- Page 8
- Speaker
- Not established
- Checked
- Sep 27, 2026