Effective Approaches to Attention-based Neural Machine Translation · HackerPapers discussions

Effective Approaches to Attention-based Neural Machine Translation — mentioned by MontyCarloHall

Thu, 08 Jan 2026 02:13:42 GMT

The confusing thing about attention in this article (and the famous "Attention is all you need" paper it's derived from) is the heavy focus on self-attention. In self-attention, Q/K/V are all derived from the same input tokens, so it's confusing to distinguish their respective purposes. I find attention much easier to understand in the original attention paper [0], which focuses on cross-attention for machine translation. In translation, the input sentence to be translated is tokenized into vectors {x_1...x_n}. The translated sentence is autoregressively generated into tokens {y_1...y_m}. To…

Paper and discussion history. A mention is not an endorsement.

Effective Approaches to Attention-based Neural Machine Translation — mentioned by lobstersammich

Fri, 24 Feb 2023 22:25:44 GMT

I think they mean Transformers in the Vaswani et al 'Attention is all you need' paper, not Generative Pretrained Transformers, specifically? Paper link below: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de... For some papers on attention mechanisms from before the 2017 'Attention is all you need' paper, check out that paper's references. Chris Manning's 2015 paper covers attention mechanisms. And so do a few other researchers from that mid-2010s time period: [21] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention based neural machine…

Paper and discussion history. A mention is not an endorsement.