Attention Is All You Need: The Transformer Paper's Five Consequential Claims
The 2017 Google Brain paper that scrapped recurrence and convolutions entirely, proposing the Transformer architecture that now underlies nearly every large language model.
1 / 5p. 2 · Yank's paraphrase
We propose the Transformer, the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
The five all come from pages 2 to 8 of 15.
We propose the Transformer, the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.
Positions the Transformer as a genuine architectural departure, not merely an incremental improvement on RNN or convolutional sequence models.
In context
…prehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 27, 28, 22]. End-to-end memory networks are based on a recurrent attention mechanism instead of sequence- aligned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks [34]. To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence- aligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models…
We propose the Transformer, is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence- aligned sequence-aligned RNNs or convolution.
Receipt
The Transformer (big) achieves 28.4 BLEU on WMT 2014 English-to-German, outperforming all previously reported models including ensembles by more than 2 BLEU, trained in 3.5 days on 8 P100 GPUs.
Anchors the paper's empirical case with a concrete state-of-the-art margin and a surprisingly modest training cost.
In context
…to be more unsure, but improves accuracy and BLEU score. 6 Results 6.1 Machine Translation On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2 . 0 BLEU, establishing a new state-of-the-art BLEU score of 28 . 4 . The configuration of this model is listed in the bottom line of Table 3. Training took 3 . 5 days on 8 P100 GPUs. Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models. On the WMT 2014 English-to-French translation task, our big model achieves a BLEU scor…
2014 English-to-German translation task, The big Transformer model (Transformer (big) in Table 2) outperforms the best achieves 28.4 BLEU on WMT 2014 English-to-German, outperforming all previously reported models including ensembles by more than 2 . 0 BLEU, trained in 3.5 days on 8 P100 GPUs.
Receipt
A self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.
Explains the architectural reason self-attention is better suited than recurrence for learning long-range dependencies.
In context
…equential operations required. The third is the path length between long-range dependencies in the network. Learning long-range dependencies is a key challenge in many sequence transduction tasks. One key factor affecting the ability to learn such dependencies is the length of the paths forward and backward signals have to traverse in the network. The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies [12]. Hence we also compare the maximum path length between any two input and output positions in networks composed of the different layer types. As noted in Table 1, a self-attention layer connects all p…
positions in networks composed of the different layer types. As noted in Table 1, A self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.
Receipt
Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
Justifies the multi-head design choice by showing what a single averaged head structurally cannot do.
In context
output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this. MultiHead( Q, K, V ) = Concat(head 1 , ..., head h ) W O where head i = Attention( QW Q i , KW K i , V W V i ) Where the projections are parameter matrices W Q i ∈ R d model × d k , W K i ∈ R d model × d k , W V i ∈ R d model × d v and W O ∈ R hd v × d model .
Receipt
Label smoothing hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
Resolves an apparent contradiction, showing a training trick that hurts one metric while genuinely improving the ones that matter for translation.
In context
For the base model, we use a rate of P drop = 0 . 1 . Label Smoothing During training, we employed label smoothing of value ϵ ls = 0 . 1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score. 6 Results 6.1 Machine Translation On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2 . 0 BLEU, establishing a new state-of-the-art BLEU score of 28 .
Label smoothing of value ϵ ls = 0 . 1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.
Receipt
Didn’t see the line you wanted? Choose your own
Make something from it
Ask a question
Ask about anything in this document. We’ll try to provide an answer.