HackerPapers

arXiv:2307.08621cs.CL

Retentive Network: A Successor to Transformer for Large Language Models

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei

Abstract

In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 6 threads, from Jul 2023 to Feb 2025. Found in 5 HN submissions from Jul 2023 to Jul 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Ask HN: Is anybody building an alternative transformer? 1 comment

    • There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15]…

      korbipView on HNParent context

  2. Eagle 7B: Soaring past Transformers 1 comment

    • These models don't have a fixed context size and are progressively fine-tuned for longer and longer contexts. The context length also doesn't impact inference cost. Another aspect of performance is not just how well does the trained model perform, but is it data efficient…

      marmadukeView on HNParent context

  3. 2023: The Year of AI 1 comment

    • For all we know, Mamba could be overhyped and could only work for small models. Remember retnet[1] which said the same thing, equal to transformer in linear time. [1]: Retentive Network: A Successor to Transformer for Large Language Models: https://arxiv.org/abs/2307.08621

      YetAnotherNickView on HNParent context

  4. Mamba: Linear-Time Sequence Modeling with Selective State Spaces 1 comment

    • > If replicated, it's arguably the first linear-time architecture with Transformer-quality performance!! RetNet was published in August (although I only learned of it last week): https://arxiv.org/abs/2307.08621

      smaddoxView on HNParent context

  5. The Eleuther AI Mafia 1 comment

    • It is important to note the claims are made for similar dataset & params. If you like some independent verification outside eleuther circle. There is microsoft paper on retnet: https://arxiv.org/abs/2307.08621 And the stanford heyena paper: https://arxiv.org/abs/2302.10866 Do…

      pico_creatorView on HNParent context

  6. Retentive Network: A Successor to Transformer Implemented in PyTorch 1 comment

    • "Retentive Network: A Successor to Transformer for Large Language Models" (2023) https://arxiv.org/abs/2307.08621 https://www.arxiv-vanity.com/papers/2307.08621/ https://huggingface.co/papers/2307.08621 https://scholar.google.com/scholar?hl=en&as_sdt=0%2C33&q=Ret...…

      westurnerView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.