HackerPapers

arXiv:2309.17453cs.CL

Efficient Streaming Language Models with Attention Sinks

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis

Abstract

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

arXiv comments
ICLR 2024
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 6 threads, from Oct 2023 to Oct 2025. Found in 3 HN submissions from Oct 2023 to Jun 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. 'Attention is all you need' coauthor says he's 'sick' of transformers 1 comment

    • There are many many problems with attention. The softmax has issues regarding attention sinks [1]. The softmax also causes sharpness problems [2]. In general this decision boundary being Euclidean dot products isn't actually optimal for everything, there are many classes of…

      kingstnapView on HNParent context

  2. Task-specific LLM evals that do and don't work 1 comment

    • Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific. Edit: The "verbosity sink" name…

      hedgehogView on HNParent context

  3. MemGPT – LLMs with self-editing memory for unbounded context 1 comment

    • Vanishing gradient was an issue for non-residual deep networks and vanilla RNNs. While the long context memory issues are along sequence dimension, not network depth. The problem could be some kind of instability of attention as it scales above 10k tokens. A recent paper…

      visargaView on HNParent context

  4. Attention Sinks in LLMs for endless fluency 1 comment

    • The paper published by Xiao et al. (2023)[0] states that "a surprisingly large amount of attention score is allocated to the initial tokens, irrespective of their relevance to the language modeling task" (p. 2). Does that mean that task prefixes used for LLM generation (e.g.…

      spidersourisView on HNParent context

  5. Training language models with pause tokens 1 comment

    • Seems related to “infinite context” by adding attention sinks. This seems to be the same mechanism at play. https://arxiv.org/abs/2309.17453.pdf

      PartiallyTypedView on HNParent context

  6. StreamingLLM: Efficient streaming technique enable infinite sequence lengths 2 comments

    • The demo [3] seems very promising. "Their method cleverly exploits the LLMs' tendency to use initial tokens as "attention sinks" to anchor the distribution of attention scores. By caching initial tokens alongside recent ones, StreamingLLM restored perplexity and achieved up to…

      TheJCDentonView on HNParent context

    • See also https://arxiv.org/abs/2309.17453, or https://news.ycombinator.com/item?id=37740932

      firebazeView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.