Neural Machine Translation by Jointly Learning to Align and Translate · HackerPapers discussions

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by cute_boi

Tue, 07 Jul 2026 19:03:23 GMT

No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/ ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385 Multi-Scale Context Aggregation by Dilated Convolutions…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by gooob

Tue, 07 Jul 2026 18:54:32 GMT

https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122 https://arxiv.org/pdf/1603.05027 https://arxiv.org/pdf/1409.2329 https://arxiv.org/pdf/1512.02595 https://arxiv.org/pdf/1409.0473 https://arxiv.org/pdf/1506.03134 https://arxiv.org/pdf/1706.03762 https://nlp.seas.harvard.edu/annotated-transformer/ https://arxiv.org/pdf/1410.5401 https://arxiv.org/pdf/1706.01427 https://arxiv.org/pdf/1806.01822…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by kavalg

Thu, 08 Jan 2026 16:42:34 GMT

Isn't the Bahdanau attention even earlier[0]? [0] https://arxiv.org/abs/1409.0473

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by spindump8930

Fri, 02 May 2025 02:33:28 GMT

You might be thinking of this article by Sebastian Ruder: https://www.ruder.io/nlp-imagenet/ Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in other words, someone with a great perspective on everything in the top article). Re: Machine Translation, note that Transformers were introduced for this task, and built on one of the earlier notions of attention in sequence models: https://arxiv.org/abs/1409.0473 (2014, 38k citations) That's not to say there weren't…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by trott

Thu, 03 Oct 2024 22:00:10 GMT

People did something similar to what you are describing 10 years ago: https://arxiv.org/abs/1409.0473 But it's trained on translations, rather than the whole Internet.

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by is_true

Sat, 18 May 2024 12:26:29 GMT

a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf https://www.cs.toronto.edu/~hinton/absps/colt93.pdf https://arxiv.org/pdf/1506.03134.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c... https://arxiv.org/pdf/1511.06391.pdf https://arxiv.org/pdf/1811.06965.pdf https://arxiv.org/pdf/1512.03385.pdf https://arxiv.org/pdf/1511.07122.pdf https://arxiv.org/pdf/1704.01212.pdf…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by sischoel

Mon, 25 Dec 2023 00:23:07 GMT

I am not sure if this was the very first paper talking about attention, but https://arxiv.org/abs/1409.0473 from 2014 is a famous one. What is still missing there, is the idea of self-attention.

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by gtoubassi

Sun, 03 Dec 2023 21:52:25 GMT

I struggled to get an intuition for this, but on another HN thread earlier this year saw the recommendation for Sebastian Raschka's series. Starting with this video: https://www.youtube.com/watch?v=mDZil99CtSU and maybe the next three or four. It was really helpful to get a sense of the original 2014 concept of attention which is easier to understand but less powerful (https://arxiv.org/abs/1409.0473), and then how it gets powerful with the more modern notion of attention. So if you have a reasonable intuition for "regular" ANNs I think this is a great place to start.

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by albertzeyer

Sun, 03 Dec 2023 20:33:40 GMT

The audience of this paper are other researchers who already know the concept of attention, which was very well known already in the field. In such research papers, such things are never explained again, as all the researchers already know this or can read other sources, which are cited, but focus on the actual research questions. In this case, the research question was simply: Can we get away by just using attention and not using the LSTM anymore? Before that, everyone was using both together. I think learning it following it more this historical development can be helpful. E.g. in this case…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by riku_iki

Sun, 19 Nov 2023 04:43:19 GMT

Attention was first proposed in this paper: https://arxiv.org/pdf/1409.0473.pdf and it is referenced from transformer paper.

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — HN submission

Mon, 03 Apr 2023 18:57:39 GMT

Neural Machine Translation by Jointly Learning to Align and Translate (2015)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by alquemist

Tue, 10 Nov 2020 13:30:47 GMT

Intelligent trial and error. 1. Transformers are an extension of the attention mechanism, a well known LSTM extension that worked well for machine translation (2014, https://arxiv.org/abs/1409.0473). A transformer model is essentially building a multi headed attention module, analogue to CNNs, then stacking several layers of them, analogue to CNNs / stacked LSTMs. (1998, http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) 2. Transfomers use residual blocks, which were introduced by the ResNet CNN arhitecture (2015, https://arxiv.org/abs/1512.03385). At the time, ResNet was topping the…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by SpaceManNabs

Fri, 28 Dec 2018 05:30:41 GMT

Seems like the article is mistaken on one part. Attention was first coined in this paper (as far as I know): https://arxiv.org/pdf/1409.0473v7.pdf The second page of the introduction of the "Hierarchical Attention Networks for Document Classification" paper mentioned in the article even cites it.

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by aglionby

Fri, 02 Mar 2018 19:45:37 GMT

Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour. [0] http://www.aclweb.org/anthology/P/P08/P08-1028 [1] http://arxiv.org/abs/1409.3215 [2] http://arxiv.org/abs/1409.0473

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by juanpino

Fri, 04 Aug 2017 08:00:16 GMT

You can read more about it in https://arxiv.org/abs/1409.0473

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by Smerity

Mon, 07 Mar 2016 06:24:51 GMT

Thanks for the great comment! For a direct link to the visual saliency maps from the paper (Figure 6) which @nicklo mentions: http://i.imgur.com/DRfaNxB.png Being able to see where a neural attention mechanism is fascinating and allows for far more than just introspection. Indeed, one of the earliest and coolest examples are neural attention mechanisms learning to align words between languages without any help[1]! http://i.imgur.com/J5zFZzN.png New papers on neural attention seem to be coming out every few weeks. I devoted much of my weekend to a naive implementation of hierarchical attentive…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — mentioned by kastnerkyle

Thu, 06 Aug 2015 21:31:02 GMT

The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3]. There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11]…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation by Jointly Learning to Align and Translate — HN submission

Sun, 25 Jan 2015 07:06:39 GMT

Neural Machine Translation by Jointly Learning to Align and Translate

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.