HackerPapers

arXiv:2305.13048cs.CL

RWKV: Reinventing RNNs for the Transformer Era

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu

Abstract

Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 9 identifiable commenters in 9 comments across 8 threads, from May 2023 to Feb 2026. Found in 1 HN submission from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. FlashAttention-T: Towards Tensorized Attention 1 comment

    • > Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear…

      cs702View on HNParent context

  2. Ask HN: Is anybody building an alternative transformer? 1 comment
  3. Mixture-of-Depths: Dynamically allocating compute in transformers 1 comment
  4. Eagle 7B: Soaring past Transformers 2 comments

    • These models don't have a fixed context size and are progressively fine-tuned for longer and longer contexts. The context length also doesn't impact inference cost. Another aspect of performance is not just how well does the trained model perform, but is it data efficient…

      marmadukeView on HNParent context

    • Here is a relevant tidbit from the RWKV paper "Limitations" section (https://arxiv.org/abs/2305.13048): First, the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae…

      jasterView on HNParent context

  5. Paving the way to efficient architectures: StripedHyena-7B 1 comment

    • Paper: https://arxiv.org/abs/2305.13048 Models (v4 are the ones from the paper): https://huggingface.co/RWKV GitHub: https://github.com/BlinkDL/RWKV-LM

      stellaathenaView on HNParent context

  6. Is anyone using self hosted LLM day to day and training it like a new employee 1 comment

    • Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work…

      filterfiberView on HNParent context

  7. Scaling Transformers to 1B Tokens 1 comment

    • > The bitter lesson [1] is going to eventually come for all of these. Eventually we'll figure out how to machine-learn the heuristic rather than hard code it. Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep…

      euclaiseView on HNParent context

  8. RWKV: Reinventing RNNs for the Transformer Era 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.