Skip to content

Sequence to Sequence Learning with Neural Networks

ArXiv: 1409.3215

🎯 Pitch

This paper introduces a groundbreaking end-to-end approach—now known as 'seq2seq'—for learning to map arbitrary-length input sequences directly to arbitrary-length output sequences using stacked LSTM neural networks as encoder and decoder. By successfully applying this method to large-scale machine translation and outperforming strong traditional baselines (and further improving them via rescoring), the authors demonstrate the power and flexibility of neural networks for general sequence transduction, opening the door to simpler, more effective solutions for tasks like translation, summarization, and beyond.

1. Executive Summary

This paper introduces a general “sequence-to-sequence” (seq2seq) method that learns to map an input sequence of arbitrary length to an output sequence of arbitrary length using two stacked Long Short-Term Memory (LSTM) networks—an encoder and a decoder. On large-scale English→French machine translation, the method, with a simple but crucial trick (reverse the source sentence), surpasses a strong phrase-based statistical machine translation baseline in direct decoding and further improves it when used for rescoring (Tables 1–2).

2. Context and Motivation

  • Problem addressed
  • Many tasks require mapping one sequence to another without a known alignment (e.g., translation, speech-to-text, question answering). Conventional deep networks expect fixed-length inputs/outputs, which does not naturally fit sequences with variable lengths (Introduction, p.1–2).

  • Prior neural approaches often required monotonic alignments (e.g., Connectionist Temporal Classification) or were used only to rescore outputs from traditional systems rather than produce translations directly (Related Work, p.7–8).

  • Why it matters

  • A domain-independent, end-to-end trainable method for arbitrary sequence transduction can collapse complex multi-stage pipelines into a single differentiable system, simplifying engineering and potentially improving performance through joint optimization (Introduction, p.2).

  • Prior approaches and limitations

  • Neural language models (RNN/Feedforward) improved translation only when rescoring outputs of phrase-based systems; they did not perform direct sequence transduction (Related Work, p.7–8).
  • Earlier encoder-decoder variants either used convolutional encoders that lose word order or struggled with long sentences (Related Work, p.7–8; Introduction noting long-range dependencies, p.2).

  • Methods that assumed monotonic alignments did not fit translation’s non-monotonic reordering (Introduction, p.2).

  • Positioning

  • The paper employs a deep, two-LSTM architecture that encodes the entire input into a fixed-size vector and then decodes the output conditioned on this vector (Fig. 1; Section 2, p.3). It introduces an unexpectedly effective preprocessing trick—reverse the source sentence—that dramatically eases optimization and boosts performance (Abstract; Sec. 2 and 3.3).

3. Technical Approach

Step-by-step description of the full system:

  • Core modeling idea (Fig. 1; Sec. 2, p.3)
  • Use one LSTM (the encoder) to read the input sequence token by token into a fixed-dimensional representation v (the final hidden state).
  • Use a second LSTM (the decoder) to generate the output sequence, one word at a time, conditioned on v and previously generated words.

  • Every sequence ends with a special end-of-sentence token <EOS> to define a probability distribution over variable-length sequences.

  • Probability formulation (Sec. 2, Eq. (1), p.3)

  • In plain language: given an input sequence x1..xT, the model first compresses it to a vector v. The probability of an output sequence y1..yT' is the product of the probabilities of each next word yt given v and the words generated so far:
    • p(y1..yT' | x1..xT) = Π_t p(yt | v, y1..y_{t-1})

  • Each next-word distribution p(yt | ...) is a softmax over the target vocabulary (Sec. 2, p.3). Softmax is the standard function that turns raw scores into a probability distribution.

  • Two separate LSTMs and depth (Sec. 2, p.3–4)

  • The encoder and decoder have separate parameters.

  • The paper finds deep LSTMs—four stacked layers—significantly outperform shallow ones (each extra layer reduced perplexity by ~10%; Sec. 3.4, p.5).

  • Key preprocessing: reverse the source sentence (Sec. 2 & 3.3, p.3–4)

  • Reverse the order of words in the source (input) sentence but not in the target.
  • Intuition: this reduces the “minimal time lag”—the average distance between corresponding source and target words—making it easier for gradients to connect input and output during training (Sec. 3.3, p.4).

  • Empirical effect: test perplexity improves from 5.8 to 4.7 and BLEU improves from 25.9 to 30.6 in single-model decoding (Sec. 3.3).

  • Training objective and procedure (Sec. 3.2 & 3.4, p.4–5)

  • Objective: maximize average log-likelihood of the correct translation log p(T|S) over sentence pairs (Sec. 3.2, p.4).
  • Optimization: stochastic gradient descent (SGD) without momentum, initial learning rate 0.7; after 5 epochs, halve every half epoch, for 7.5 epochs total (Sec. 3.4).
  • Initialization: uniform in [-0.08, 0.08] (Sec. 3.4).
  • Gradient clipping: L2-norm capped at 5 to handle exploding gradients (Sec. 3.4).
  • Mini-batching: batch size 128; sentences in a batch are bucketed by similar lengths for ~2× speedup (Sec. 3.4).

  • Vocabularies: source 160k, target 80k most frequent words; out-of-vocabulary words become UNK (Sec. 3.1). This means outputs containing any word outside the 80k target vocabulary are forced to UNK.

  • Architecture scale and embeddings (Sec. 3.4 & Abstract)

  • 4-layer encoder and 4-layer decoder; each layer has 1000 cells; word embeddings are 1000-dimensional (Sec. 3.4).
  • Each LSTM has an 8,000-dimensional state when considering multi-layer states (Abstract; Sec. 3.4).
  • Total parameters: ~384 million, with ~64 million in recurrent connections (Sec. 3.4).

  • Output softmax over 80k words is computed naively (no hierarchical or sampled softmax), contributing to computational load (Sec. 3.4).

  • Decoding (Sec. 3.2, p.4)

  • Use left-to-right beam search with beam size B (typical values: 1, 2, 12 in experiments). Beam search keeps the B most probable partial sequences at each step; once <EOS> is produced, a hypothesis is complete and removed from the beam.

  • Observation: even B=1 performs surprisingly well; B=2 captures most of the gains (Table 1).

  • Rescoring setup (Sec. 3.2, p.4)

  • For the same dataset, use the model to assign log-probabilities to each hypothesis in an SMT system’s 1000-best list, and average that score with the SMT score to rerank (Table 2).

  • Hardware and parallelization (Sec. 3.5, p.5)

  • Implementation across 8 GPUs: four GPUs for the four LSTM layers (pipelined) and four GPUs to shard the softmax (each handles a 1000×20000 matrix multiplication).

  • Throughput: ~6,300 words/s with batch size 128 (both languages combined). Training ~10 days (Sec. 3.5).

  • Dataset and metric (Sec. 3.1 & 3.6, p.4–5)

  • Data: WMT’14 English→French, 12M sentence pairs (304M English, 348M French words) from a selected, cleaned subset (Sec. 3.1).
  • Metric: cased BLEU using multi-bleu.pl on tokenized outputs (Sec. 3.6). BLEU is an n-gram precision-based metric (0–100) commonly used for translation quality.

4. Key Insights and Innovations

  • A deep encoder–decoder LSTM that performs direct sequence-to-sequence learning without explicit alignment (Fig. 1; Sec. 2)
  • What’s new: Uses a multi-layer LSTM to compress an entire input sentence into a fixed vector and decode conditioned on it, achieving strong direct translation quality on a large-scale benchmark (Table 1).

  • Why it matters: Demonstrates that a purely neural, end-to-end model can directly outperform a phrase-based SMT baseline (Table 1), moving beyond using neural nets merely as rescoring components.

  • Source-reversal trick that reduces effective time lag and makes optimization easier (Sec. 3.3)

  • What’s new: Reverse input sentences during training and inference (decoder still generates forward). This is a data transformation, not an architectural change.
  • Impact: Large gains in both perplexity (5.8 → 4.7) and BLEU (25.9 → 30.6) for single-model decoding (Sec. 3.3). It also improves performance on long sentences (Fig. 3, left), contradicting concerns about fixed-vector encoders failing on long inputs.

  • Significance: A simple, low-cost change that addresses the hardest part of training sequence transducers—credit assignment over long distances.

  • Depth and large capacity are important (Sec. 3.4)

  • Finding: Four-layer LSTMs “significantly” outperform shallow variants; each additional layer cuts perplexity by nearly 10% (Sec. 3.4).

  • Significance: Establishes the value of depth for sequence modeling and justifies the computational investment.

  • Emergent sentence representations sensitive to word order yet robust to voice (Fig. 2; Sec. 3.8)

  • Evidence: 2D PCA on hidden states shows phrases cluster by meaning and preserve order differences while being relatively invariant to active vs. passive constructions (Fig. 2).

  • Significance: Indicates that the encoder learns non-bag-of-words sentence representations helpful for downstream mapping.

  • Practical decoding insight: small beams suffice (Table 1)

  • Observation: An ensemble of 5 reversed LSTMs with beam size 2 nearly matches beam 12 (34.50 vs. 34.81 BLEU; Table 1).
  • Significance: Reduces decoding cost without major quality loss.

5. Experimental Analysis

  • Evaluation methodology
  • Dataset: WMT’14 En→Fr, 12M sentence pairs (Sec. 3.1).
  • Metrics: cased BLEU via multi-bleu.pl (Sec. 3.6).
  • Baseline: Public phrase-based SMT system (Schwenk, 2014), 33.30 BLEU (Sec. 3.6; Table 1).

  • Setups tested:

    • Direct decoding with varying beam sizes and ensembles (Table 1).
    • Rescoring SMT 1000-best lists with single and ensemble LSTMs (Table 2).
    • Analysis of reversing input sentences (Sec. 3.3).
    • Performance vs. sentence length and lexical rarity (Fig. 3).
    • Qualitative inspection of long-sentence translations (Table 3).
    • Representation visualization (Fig. 2).

  • Main quantitative results

  • Direct decoding (Table 1):
    • “Single forward LSTM, beam 12”: 26.17 BLEU.
    • “Single reversed LSTM, beam 12”: 30.59 BLEU.
    • “Ensemble of 5 reversed LSTMs, beam 2”: 34.50 BLEU.
    • “Ensemble of 5 reversed LSTMs, beam 12”: 34.81 BLEU.
    • Baseline SMT: 33.30 BLEU.
    • Quote:

      Table 1 … Ensemble of 5 reversed LSTMs, beam size 12: 34.81; Baseline System: 33.30.

  • Rescoring (Table 2):
    • Baseline SMT: 33.30 BLEU.
    • “Rescore 1000-best with single forward LSTM”: 35.61.
    • “Rescore 1000-best with single reversed LSTM”: 35.85.
    • “Rescore 1000-best with ensemble of 5 reversed LSTMs”: 36.5.
    • Quote:

      Table 2 … Ensemble rescoring achieves 36.5 BLEU; best WMT’14 system listed: 37.0.

  • Effect of reversing input (Sec. 3.3):
    • Quote:

      “test perplexity dropped from 5.8 to 4.7, and … BLEU increased from 25.9 to 30.6.”

  • Robustness to sentence length and rare words (Fig. 3):

    • Left panel: no degradation for sentences <35 words; only minor degradation for the longest sentences.
    • Right panel: performance declines as average word rarity increases, but remains competitive with the baseline (Fig. 3).

  • Qualitative results (Table 3; Sec. 3.7)

  • Long sentences (often >30 words): translations are structurally coherent and semantically close to references, with remaining errors largely tied to UNK substitutions due to the 80k target vocabulary cap (e.g., named entities marked as UNK).

  • Do the experiments support the claims?

  • Yes, for three central claims:
    • Encoder–decoder LSTMs can perform direct large-scale translation competitively: 34.81 BLEU surpasses the 33.30 phrase-based baseline (Table 1).
    • Source reversal materially simplifies learning and improves accuracy, corroborated by both perplexity and BLEU gains (Sec. 3.3; Table 1).
    • The fixed-vector encoder can handle long sentences with the reversal trick, as seen in Fig. 3 and Table 3.
  • Ablations/robustness:
    • Forward vs. reversed input is a clear ablation (Tables 1–2).
    • Beam-size effects are explored (Table 1).
    • Ensemble effects are measured (Table 1).
  • Caveats:
    • BLEU comparisons to the best WMT’14 system depend on evaluation scripts; the paper notes a discrepancy (Sec. 3.6).

6. Limitations and Trade-offs

  • Fixed-size sentence representation

  • Assumption: compressing an entire source sentence into a single vector v suffices. While reversal helps, this can, in principle, bottleneck information flow for very long or complex sentences (Sec. 2; Fig. 3 shows only minor degradation but still some drop for the longest cases).

  • Vocabulary cap and UNK handling

  • Target vocabulary limited to 80k; any out-of-vocabulary word becomes UNK (Sec. 3.1).

  • Impact: BLEU is “penalized on out-of-vocabulary words” (Abstract). Qualitative examples show UNK mainly for names and rare terms (Table 3).

  • Computational demands

  • Large model (384M parameters) with naive 80k-word softmax; training required 8 GPUs for ~10 days (Sec. 3.4–3.5).

  • Beam search still required for best decoding (though small beams suffice) (Table 1).

  • Data requirements and scope

  • Results hinge on very large, clean parallel corpora (12M sentence pairs; Sec. 3.1).

  • Experiments focus on a single language pair (En→Fr). Generality is argued but not empirically verified across diverse language pairs or domains (Sec. 3.6).

  • Simplicity of rescoring combination

  • Rescoring uses an equal-weight average of SMT and LSTM scores without weight tuning, which may underutilize the complementary strengths or complicate fairness of comparisons (Sec. 3.2).

7. Implications and Future Directions

  • Field impact

  • Demonstrates that a pure neural encoder–decoder can directly perform sequence transduction at scale, surpassing a strong phrase-based SMT baseline (Table 1). This establishes seq2seq as a practical alternative to hand-engineered pipelines and opens the door to end-to-end learning for many sequence tasks (Introduction; Conclusion).

  • Practical applications

  • Beyond translation: speech recognition, text summarization, dialogue systems, code generation, and any task mapping input sequences to output sequences without known alignments (Introduction, p.2).

  • Follow-up research directions (grounded in observed gaps)

  • Address the fixed-vector bottleneck by enabling the decoder to access encoder states dynamically (the paper references attention-style ideas in Related Work, p.7–8).
  • Improve handling of rare and unseen words to reduce UNK penalization—e.g., subword or character-level modeling, or copying mechanisms (Limitation noted in Abstract and Sec. 3.1).
  • Reduce training cost of large-vocabulary softmax (sampled/approximate softmax) and explore more efficient parallelism beyond layer-wise GPU pipelining (Sec. 3.5).
  • Systematic evaluation across multiple language pairs, domains, and resource regimes (low-resource settings) to test generality (Sec. 3.6 scope).
  • Analyze and leverage the learned sentence representations (v) for downstream tasks; Fig. 2 suggests they encode meaningful structure beyond bag-of-words (Sec. 3.8).

Key citations to the paper’s content: - Architecture and modeling: Fig. 1; Sec. 2 (p.3–4); Eq. (1). - Data and setup: Sec. 3.1–3.2 (p.4). - Training/implementation: Sec. 3.4–3.5 (p.5). - Main results: Tables 1–2 (p.5–6). - Reversal analysis: Sec. 3.3 (p.4). - Length/rarity robustness: Fig. 3 (p.7). - Representation analysis: Fig. 2 (p.6). - Qualitative outputs: Table 3 (p.7). - Context/related work: Intro (p.1–3), Related Work (p.7–8). - Summary significance: Abstract; Conclusion (p.8).