On The Computational Complexity of Self-Attention · HackerPapers discussions

On The Computational Complexity of Self-Attention — mentioned by strommec

Mon, 26 Oct 2026 23:39:46 GMT

It's surprising that Exponential Time Hypothesis would be false. Quoting a paper from 2022[1]: > We prove that the time complexity of self-attention is necessarily quadratic in the input length, unless the Strong Exponential Time Hypothesis (SETH) is false[...] If OpenAI's proof is correct, does that mean self-attention can be implemented in sub-quadratic time? I.e. we could eventually have all the most advanced LLMs running on cheap hardware? To put it more playfully, is ETH being false actually bearish for Nvidia? [1]: https://arxiv.org/abs/2209.04881

Paper and discussion history. A mention is not an endorsement.

On The Computational Complexity of Self-Attention — mentioned by noosphr

Mon, 12 Jan 2026 00:25:34 GMT

https://arxiv.org/abs/2209.04881

Paper and discussion history. A mention is not an endorsement.

On The Computational Complexity of Self-Attention — mentioned by logicchains

Wed, 10 Apr 2024 13:42:58 GMT

Transformers are special because theoretically you can't make something "weaker" than a transformer without it losing some accuracy: https://arxiv.org/abs/2209.04881 .

Paper and discussion history. A mention is not an endorsement.

On The Computational Complexity of Self-Attention — mentioned by logicchains

Sun, 31 Mar 2024 07:06:18 GMT

>Lately I've been wondering... is this a problem, or a strength? It's a strength; fundamentally it's impossible to achieve the same degree of accuracy with a sub-quadratic attention mechanism: https://arxiv.org/abs/2209.04881 (unless the Strong Exponential Time Hypothesis is false, which is very unlikely, like P=NP).

Paper and discussion history. A mention is not an endorsement.

On The Computational Complexity of Self-Attention — mentioned by logicchains

Tue, 05 Mar 2024 05:45:12 GMT

There's fundamentally a trade-off: https://arxiv.org/abs/2209.04881 . (Unless the Strong Exponential Time Hypothesis is false, but that's pretty unlikely, like P being equal to NP).

Paper and discussion history. A mention is not an endorsement.

On The Computational Complexity of Self-Attention — mentioned by nyrikki

Sun, 25 Feb 2024 19:28:27 GMT

Strong Exponential Time Hypothesis Here is how it relates to attention. https://arxiv.org/abs/2209.04881

Paper and discussion history. A mention is not an endorsement.