Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence · HackerPapers discussions

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence — mentioned by cs702

Wed, 04 Feb 2026 00:28:22 GMT

> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.

Paper and discussion history. A mention is not an endorsement.

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence — mentioned by jasonjmcghee

Wed, 08 May 2024 15:30:06 GMT

They reference "a GPT-3 model with 356M parameters" So GPT-3 Medium (from the GPT-3 paper) - feels pretty disingenuous to list that as no one is referencing that model when they say "GPT-3", but the 175B model. I wasn't aware that size of the model (356M) was released- what am I missing here? I also think it's relatively well understood that (with our current methods) transformers have a tipping point with parameter count, and I don't know of any models less than ~3B that are useful- arguably 7B. Compare these benchmarks to, say, the RWKV 5/6 paper https://arxiv.org/abs/2404.05892

Paper and discussion history. A mention is not an endorsement.

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence — HN submission

Wed, 10 Apr 2024 10:56:11 GMT

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence — HN submission

Wed, 10 Apr 2024 02:23:50 GMT

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.