Gated Delta Networks: Improving Mamba2 with Delta Rule · HackerPapers discussions

Gated Delta Networks: Improving Mamba2 with Delta Rule — mentioned by mynti

Fri, 12 Sep 2025 08:40:49 GMT

For anyone curious about what the Gated Delta Network is: https://arxiv.org/pdf/2412.06464

Paper and discussion history. A mention is not an endorsement.

Gated Delta Networks: Improving Mamba2 with Delta Rule — mentioned by juank10

Tue, 17 Jun 2025 07:48:38 GMT

Funnily enough, the code was deleted in the repo, but can still be seen in the commits. It's what you would expect from the paper :D On the general topic of non-attention LLMs, I recommend checking out the MesaNet [1], Rodimus [2], Gated DeltaNet [3], or Mamba2 [4]. They are currently SOTA. However, I have yet to see a compelling non attention based model that achieves good performance on code, math, reasoning, or multi-turn QA tasks. I do not think we are getting rid of attention soon, I believe the ability to look back is crucial in certain tasks. [1] https://arxiv.org/abs/2506.05233 [2]…

Paper and discussion history. A mention is not an endorsement.

Gated Delta Networks: Improving Mamba2 with Delta Rule — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.