Low-Rank Bottleneck in Multi-head Attention Models — mentioned by namibj

Tue, 15 Jul 2025 12:55:34 GMT

It's inherent, see https://arxiv.org/abs/2002.07028 (as I detailed in my sibling comment to yours just now, but before I saw yours). That said, there are architecture sizing ways that allow much better long-context performance at the cost of some short-context performance for a given parameter count and inference compute budget.

Paper and discussion history. A mention is not an endorsement.

Low-Rank Bottleneck in Multi-head Attention Models — mentioned by namibj

Tue, 15 Jul 2025 12:48:30 GMT

On this note I want to point at "Low-Rank Bottleneck in Multi-head Attention Models", which details how attention inherently needs the query dimension to match or exceed the sequence length to allow precise (and especially, sharp) targeting. It may be that dimension-starved pretrained transformer models rely heavily on context being correctly "tagged" in all relevant aspects the very instant it's inserted into the KV cache, e.g. necessitating negation to be prefixed to a fact instead of allowing post-fix negation. The common LLM chat case is telling the model it just spewed…

Paper and discussion history. A mention is not an endorsement.

Low-Rank Bottleneck in Multi-head Attention Models · HackerPapers discussions

Low-Rank Bottleneck in Multi-head Attention Models — mentioned by namibj

Low-Rank Bottleneck in Multi-head Attention Models — mentioned by namibj