RWKV: Reinventing RNNs for the Transformer Era · HackerPapers discussions

RWKV: Reinventing RNNs for the Transformer Era — mentioned by cs702

Wed, 04 Feb 2026 00:28:22 GMT

> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by htrp

Fri, 14 Feb 2025 20:40:57 GMT

Anyone know what the rwkv people are up to now? https://arxiv.org/abs/2305.13048

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by conradev

Sun, 07 Apr 2024 21:08:33 GMT

Yep: https://arxiv.org/abs/2305.13048

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by jaster

Mon, 29 Jan 2024 09:18:39 GMT

Here is a relevant tidbit from the RWKV paper "Limitations" section (https://arxiv.org/abs/2305.13048): First, the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to…

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by marmaduke

Mon, 29 Jan 2024 06:16:40 GMT

These models don't have a fixed context size and are progressively fine-tuned for longer and longer contexts. The context length also doesn't impact inference cost. Another aspect of performance is not just how well does the trained model perform, but is it data efficient (performance per token trained)? The comparison with Pythia (an open GPT) is shown in the article. The rwkv4 paper is quite detailed and has examples of prompt and responses on the last few pages https://arxiv.org/abs/2305.13048 And iirc rwkv5 is very similar to retnet which is detailed here https://arxiv.org/abs/2307.08621…

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by stellaathena

Sat, 09 Dec 2023 00:34:10 GMT

Paper: https://arxiv.org/abs/2305.13048 Models (v4 are the ones from the paper): https://huggingface.co/RWKV GitHub: https://github.com/BlinkDL/RWKV-LM

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by filterfiber

Thu, 30 Nov 2023 19:13:01 GMT

Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC. Here's a decent overview https://aman.ai/primers/ai/context-length-extension/ Rope…

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by euclaise

Thu, 06 Jul 2023 15:20:05 GMT

> The bitter lesson [1] is going to eventually come for all of these. Eventually we'll figure out how to machine-learn the heuristic rather than hard code it. Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep sequences. Linear RNNs and RWKV are examples of RNNs on deep sequences: https://arxiv.org/abs/2303.06349 https://arxiv.org/abs/2305.13048

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — mentioned by elromulous

Tue, 23 May 2023 02:35:38 GMT

https://arxiv.org/pdf/2305.13048 ^direct pdf link

Paper and discussion history. A mention is not an endorsement.

RWKV: Reinventing RNNs for the Transformer Era — HN submission

Tue, 23 May 2023 01:50:00 GMT

RWKV: Reinventing RNNs for the Transformer Era

358 points; 171 discussion replies on HN. Paper and source history. A submission is not an endorsement.