Massive Exploration of Neural Machine Translation Architectures · HackerPapers discussions

Massive Exploration of Neural Machine Translation Architectures — mentioned by ansk

Thu, 17 Oct 2024 22:13:51 GMT

Key-based attention is not attributable to the Transformer paper. First paper I can find where keys, queries, and values are distinct matrices is https://arxiv.org/abs/1703.03906, described at the end of section 2. The authors of the Transformer paper are very clear in how they describe their contribution to the attention formulation, writing "Dot-product attention is identical to our algorithm, except for the scaling factor". I think it's fair to state that multi-head is the paper's only substantial contribution to the design of attention mechanisms. I think you're overestimating the degree…

Paper and discussion history. A mention is not an endorsement.

Massive Exploration of Neural Machine Translation Architectures — mentioned by msamwald

Sat, 10 Jun 2017 08:29:26 GMT

A recent paper exploring various variatons of neural machine translation architectures found that LSTMs consistently outperformed GRUs in that task. https://arxiv.org/abs/1703.03906

Paper and discussion history. A mention is not an endorsement.

Massive Exploration of Neural Machine Translation Architectures — mentioned by visarga

Fri, 24 Mar 2017 04:46:07 GMT

TL;DR - It's the code from "Massive Exploration of Neural Machine Translation Architectures", where they run extensive hyperparameter search using 250K hours of GPU time. Direct GitHub link: https://github.com/google/seq2seq Arxiv: https://arxiv.org/abs/1703.03906

Paper and discussion history. A mention is not an endorsement.