Fast Attention Requires Bounded Entries — mentioned by thomasahle

Wed, 04 Feb 2026 15:33:26 GMT

There's a graveyard of 100s of papers with "approximate near linear time attention." They always hope the speed increase makes up for the lower quality, but it never does. The quadratic time seems inherent to the problem. Indeed, there are lower bounds showing that sub n^2 algorithms can't work: https://arxiv.org/pdf/2302.13214

Paper and discussion history. A mention is not an endorsement.

Fast Attention Requires Bounded Entries · HackerPapers discussions

Fast Attention Requires Bounded Entries — mentioned by thomasahle