Efficient Streaming Language Models with Attention Sinks · HackerPapers discussions

Efficient Streaming Language Models with Attention Sinks — mentioned by kingstnap

Fri, 24 Oct 2025 19:48:46 GMT

There are many many problems with attention. The softmax has issues regarding attention sinks [1]. The softmax also causes sharpness problems [2]. In general this decision boundary being Euclidean dot products isn't actually optimal for everything, there are many classes of problem where you want polyhedral cones [3]. Positional embedding are also janky af and so is rope tbh, I think Cannon layers are a more promising alternative for horizontal alignment [4]. I still think there is plenty of room to improve these things. But a lot of focus right now is unfortunately being spent on…

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — HN submission

Fri, 06 Jun 2025 19:32:08 GMT

Efficient Streaming Language Models with Attention Sinks

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — mentioned by hedgehog

Mon, 09 Dec 2024 21:37:26 GMT

Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific. Edit: The "verbosity sink" name is inspired by the idea from the paper below although they're not actually at all the same thing. https://arxiv.org/abs/2309.17453

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — mentioned by visarga

Mon, 16 Oct 2023 21:49:52 GMT

Vanishing gradient was an issue for non-residual deep networks and vanilla RNNs. While the long context memory issues are along sequence dimension, not network depth. The problem could be some kind of instability of attention as it scales above 10k tokens. A recent paper suggests attention mechanism needs a default value (a "sink"), and its absence produces instability. https://arxiv.org/abs/2309.17453 Another paper says the middle part is lossy while the beginning and end are better attended.

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — mentioned by spidersouris

Mon, 09 Oct 2023 18:37:27 GMT

The paper published by Xiao et al. (2023)[0] states that "a surprisingly large amount of attention score is allocated to the initial tokens, irrespective of their relevance to the language modeling task" (p. 2). Does that mean that task prefixes used for LLM generation (e.g. "translate: [sentence]") are actually attention sinks? Or are they not? I don't really understand what they mean by "irrespective of their relevance to the language modeling task." [0] https://arxiv.org/pdf/2309.17453.pdf

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — mentioned by PartiallyTyped

Wed, 04 Oct 2023 18:16:54 GMT

Seems related to “infinite context” by adding attention sinks. This seems to be the same mechanism at play. https://arxiv.org/abs/2309.17453.pdf

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — mentioned by firebaze

Tue, 03 Oct 2023 20:13:06 GMT

See also https://arxiv.org/abs/2309.17453, or https://news.ycombinator.com/item?id=37740932

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — mentioned by TheJCDenton

Tue, 03 Oct 2023 20:10:46 GMT

The demo [3] seems very promising. "Their method cleverly exploits the LLMs' tendency to use initial tokens as "attention sinks" to anchor the distribution of attention scores. By caching initial tokens alongside recent ones, StreamingLLM restored perplexity and achieved up to 22x faster decoding than prior techniques." [1] "We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more." [2] "we discover that adding a placeholder token as a dedicated attention sink during pre-training can further…

Paper and discussion history. A mention is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — HN submission

Tue, 03 Oct 2023 20:10:46 GMT

StreamingLLM: Efficient streaming technique enable infinite sequence lengths

118 points; 12 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Efficient Streaming Language Models with Attention Sinks — HN submission

Mon, 02 Oct 2023 17:35:05 GMT

Efficient Streaming Language Models with Attention Sinks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.