Linformer: Self-Attention with Linear Complexity · HackerPapers discussions

Linformer: Self-Attention with Linear Complexity — mentioned by ctoth

Sat, 22 Aug 2020 19:46:12 GMT

I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like to mention is that GPT-3 uses exactly none of the new techniques that have been coming out in the last two years that would have significant impact on text generation. From working methods to apply GANs to text, to far more efficient transformer models that can handle longer sequences. For instance [1] [2] [3] for better direction, or [4] [5] [6] for efficiency. Or perhaps the outside view might help. After seeing GPT-2 last year, did…

Paper and discussion history. A mention is not an endorsement.

Linformer: Self-Attention with Linear Complexity — mentioned by 101101001010

Fri, 03 Jul 2020 12:45:46 GMT

They are also efficient at inference-time. On GPUs the difference is noticeable only for sequences of length > 1024 (2048 for reformer since it adds some operations for hashing) thanks to the massive parallelism of GPUs amortizing the quadratic effect of the "usual" self-attention mechanism. [edit] Linformer (https://arxiv.org/pdf/2006.04768.pdf) is a different project from the one linked in https://linear-transformers.com/ (Transformers are RNNs https://arxiv.org/pdf/2006.16236.pdf).

Paper and discussion history. A mention is not an endorsement.

Linformer: Self-Attention with Linear Complexity — HN submission

Wed, 10 Jun 2020 20:23:39 GMT

Linformer: Self-Attention with Linear Complexity

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.