HackerPapers

arXiv:1409.0473cs.CL

Neural Machine Translation by Jointly Learning to Align and Translate

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

Abstract

Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.

arXiv comments
Accepted at ICLR 2015 as oral presentation
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 16 identifiable commenters in 16 comments across 14 threads, from Aug 2015 to Jul 2026. Found in 2 HN submissions from Jan 2015 to Apr 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments14 threads

  1. 30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format 2 comments

    • https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122…

      gooobView on HNParent context

    • No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks…

      cute_boiView on HNParent context

  2. The Q, K, V Matrices 1 comment
  3. When ChatGPT broke the field of NLP: An oral history 1 comment

    • You might be thinking of this article by Sebastian Ruder: https://www.ruder.io/nlp-imagenet/ Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in…

      spindump8930View on HNParent context

  4. Were RNNs all we needed? 1 comment

    • People did something similar to what you are describing 10 years ago: https://arxiv.org/abs/1409.0473 But it's trained on translations, rather than the whole Internet.

      trottView on HNParent context

  5. Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters” 1 comment

    • a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf…

      is_trueView on HNParent context

  6. "Attention", "Transformers", in Neural Network "Large Language Models" 1 comment

    • I am not sure if this was the very first paper talking about attention, but https://arxiv.org/abs/1409.0473 from 2014 is a famous one. What is still missing there, is the idea of self-attention.

      sischoelView on HNParent context

  7. LLM Visualization 2 comments

    • The audience of this paper are other researchers who already know the concept of attention, which was very well known already in the field. In such research papers, such things are never explained again, as all the researchers already know this or can read other sources, which…

      albertzeyerView on HNParent context

    • I struggled to get an intuition for this, but on another HN thread earlier this year saw the recommendation for Sebastian Raschka's series. Starting with this video: https://www.youtube.com/watch?v=mDZil99CtSU and maybe the next three or four. It was really helpful to get a…

      gtoubassiView on HNParent context

  8. Google is not a tech company 1 comment

    • Attention was first proposed in this paper: https://arxiv.org/pdf/1409.0473.pdf and it is referenced from transformer paper.

      riku_ikiView on HNParent context

  9. A Primer in BERTology: What We Know About How Bert Works 1 comment

    • Intelligent trial and error. 1. Transformers are an extension of the attention mechanism, a well known LSTM extension that worked well for machine translation (2014, https://arxiv.org/abs/1409.0473). A transformer model is essentially building a multi headed attention module…

      alquemistView on HNParent context

  10. What Kagglers Are Using for Text Classification 1 comment

    • Seems like the article is mistaken on one part. Attention was first coined in this paper (as far as I know): https://arxiv.org/pdf/1409.0473v7.pdf The second page of the introduction of the "Hierarchical Attention Networks for Document Classification" paper mentioned in the…

      SpaceManNabsView on HNParent context

  11. Understanding word vectors 1 comment

    • Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've…

      aglionbyView on HNParent context

  12. Transitioning entirely to neural machine translation 1 comment
  13. Dynamic Memory Networks for Visual and Textual Question Answering 1 comment

    • Thanks for the great comment! For a direct link to the visual saliency maps from the paper (Figure 6) which @nicklo mentions: http://i.imgur.com/DRfaNxB.png Being able to see where a neural attention mechanism is fascinating and allows for far more than just introspection.…

      SmerityView on HNParent context

  14. A Beginner’s Guide to Restricted Boltzmann Machines 1 comment

    • The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs…

      kastnerkyleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.