A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond — mentioned by mcbuilder

Tue, 10 Oct 2023 19:17:44 GMT

The transformer model was invented to attend to context over the entire sequence length. Look at how the original authors used the Transformer for NMT in the original Vaswani et al publication. https://github.com/jadore801120/attention-is-all-you-need-py... See how the source sentence is run through the encoded, and then the decoder is run in an auto-regressive mode. It's been a while, so I can't recall if the encoded embeddings are masked to the same index to the decoder input sequence, and I don't see why you would even need to do particular task auto-regressively in theory. Found this…

Paper and discussion history. A mention is not an endorsement.

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond · HackerPapers discussions

A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond — mentioned by mcbuilder