HackerPapers

arXiv:2404.07143cs.CL

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal

Abstract

This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

arXiv comments
9 pages, 4 figures, 4 tables (v2 adds: background, implementation details, recent citations and acknowledgments)
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 4 identifiable commenters in 6 comments across 6 threads, from Apr 2024 to May 2024. Found in 3 HN submissions from Apr 2024 to Apr 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. New attention mechanisms that outperform standard multi-head attention 1 comment

    • Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs. [1] https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba... "You Need to Pay…

      westurnerView on HNParent context

  2. Mixtral 8x22B 1 comment

    • tons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too: https://arxiv.org/html/2404.08801v1 Meta Megalodon https://arxiv.org/html/2404.07143v1 Google Infini-Attention https://arxiv.org/html/2402.13753v1 LongRoPE and a ton more

      underlinesView on HNParent context

  3. Ask HN: Is RAG the Future of LLMs? 1 comment

    • https://arxiv.org/html/2404.07143v1 “This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”

      ttulView on HNParent context

  4. Google's new technique gives LLMs infinite context 1 comment

    • "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (2024) https://arxiv.org/abs/2404.07143 - https://news.ycombinator.com/item?id=40001626#40020560

      westurnerView on HNParent context

  5. Leave No Context Behind: Efficient Infinite Context Transformers 1 comment

    • "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (2024) https://arxiv.org/abs/2404.07143 : > This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and…

      westurnerView on HNParent context

  6. Scaling will never get us to AGI 1 comment

    • Scaling alone will certainly not get us to AGI, but that's trivially obvious. The "scale is all you need" argument, from anyone intelligent, assumes that other obvious deficiencies such as lack of short-term memory (to enable more capable planning/reasoning) will also be taken…

      HarHarVeryFunnyView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.