Mamba: Linear-Time Sequence Modeling with Selective State Spaces · HackerPapers discussions

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by cs702

Wed, 04 Feb 2026 00:28:22 GMT

> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by godelski

Fri, 08 Aug 2025 06:20:57 GMT

> Something like GPT-2 I got 2 things to say here 1) There's plenty of things that can achieve similar performance to GPT-2 these days. We mentioned Mamba, they compared to GPT-3 in their first paper[0]. They compare with the open sourced version and you'll also see some other architectures referenced there like Hyena and H3. It's the GPT-Neo and GPT-J models. Remember GPT-3 is pretty much just a scaled up GPT-2. 2) I think you are underestimating the costs to train some of these things. I know Karpathy said you can now train GPT-2 for like $1k[1] but a single training run is a small portion…

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by bob1029

Sat, 31 May 2025 08:45:37 GMT

I think it is too risky to build a company around the premise that someone won't soon solve the quadratic scaling issue. Especially, when that company involves creating ASICs. E.g.: https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by viktor_von

Sat, 05 Oct 2024 20:08:45 GMT

> The information cost of making the RNN state way bigger is high when done naively, but maybe someone can figure out a clever way to avoid storing full hidden states in memory during training or big improvements in hardware could make memory use less of a bottleneck. Isn't this essentially what Mamba [1] does via its 'Hardware-aware Algorithm'? [1] https://arxiv.org/pdf/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by simonw

Tue, 16 Jul 2024 18:38:19 GMT

There's a paper: https://arxiv.org/abs/2312.00752 I haven't seen any good non-paper explainers yet.

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by Fripplebubby

Mon, 01 Jul 2024 16:38:15 GMT

To further problematize this question (which I don't feel like I can actually answer), consider this paper: "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention" - https://arxiv.org/pdf/2006.16236 What this shows is that actually a specific narrow definition of transformer (a transformer with "causal masking" - see paper) is equivalent to an RNN, and vice versa. Similarly Mamba (https://arxiv.org/abs/2312.00752), the other hot architecture at the moment, has an equivalent unit to a gated RNN. For performance reasons, I believe they use an equivalent CNN during…

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by edflsafoiewq

Sat, 08 Jun 2024 16:29:43 GMT

The doc comment at the top of the .py file is sufficiently descriptive """Simple, minimal implementation of Mamba in one file of Numpy adapted from (1) and inspired from (2). Suggest reading the following before/while reading the code: [1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Albert Gu and Tri Dao) https://arxiv.org/abs/2312.00752 [2] The Annotated S4 (Sasha Rush and Sidd Karamcheti) https://srush.github.io/annotated-s4

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by sva_

Sat, 08 Jun 2024 09:54:57 GMT

It's an LLM architecture competing with transformers: https://arxiv.org/abs/2312.00752 Proponents of it usually highlight it's inference performance, in particular linear scaling with the input tokens.

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by fancyfredbot

Mon, 04 Mar 2024 16:50:42 GMT

A paper describing how you might do it published in December last year. The paper was "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". To be clear I don't know if Claude and Gemini actually use this technique but I would not be surprised if they did something similar: https://arxiv.org/abs/2312.00752 https://github.com/state-spaces/mamba

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by yuhongsun

Sat, 02 Mar 2024 22:01:15 GMT

Ya, a lot of knowledge is locked away in metadata, how documents are structured, in non-textual representations, or sometimes even just in the head of experts. We definitely want to be more than a shallow access to documents and we're building with that in mind. We're currently working to include more metadata and organizational understanding, with plans to tackle OCR, NL-to-SQL, code search and knowledge graphs in the future. While we don't have concrete dates for all of these, hopefully this gives some visibility into our vision at least! Regarding the context size, the research community…

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by swimwiththebeat

Fri, 01 Mar 2024 23:11:25 GMT

Does anyone know if this is using the Mamba architecture[1] instead of transformers? It looks like it uses a state space model (SSM) layer. [1]: https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by jackcook

Fri, 23 Feb 2024 21:04:05 GMT

Yes you're spot on, the nonlinearities come from the full Mamba blocks, which I left out of this post for simplicity/to focus on the bigger ideas the paper introduced. You can see it marked by the "X" on the right-most part of Figure 3 in the Mamba paper: https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by jxmorris12

Fri, 23 Feb 2024 18:34:50 GMT

In case people are wondering why Mamba is exciting: There's this idea in AI right now that "scaling" models to be bigger and train on more data always makes them better. This has led to a science of "scaling laws" which study just how much bigger models need to be and how much data we need to train them on to make them a certain amount better. The relationship between model size, training data size, and performance turns out to be quite predictable. Transformers are great because they can continue scaling and giving us better performance – unlike, we think, RNNs. Probably the most exciting…

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by TweedBeetle

Thu, 15 Feb 2024 18:47:30 GMT

Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture. Here’s the paper: https://arxiv.org/abs/2312.00752 And here’s a great podcast episode on it: https://www.cognitiverevolution.ai/emergency-pod-mamba-memor...

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by sanxiyn

Mon, 29 Jan 2024 06:56:48 GMT

I agree that on balance, we should spend more effort on data than modeling, but it is just not true that modeling doesn't matter. Transformer-2023 is different from Trnasformer-2020 and cumulative improvement is significant. https://arxiv.org/abs/2312.00752 did such benchmark. If choice between Transformer and RWKV doesn't seem to matter to you, the only reason is that while Transformer-2020 evolved to Transformer-2023, RWKV-v1 (which is from 2021) also evolved to RWKV-v5. If you use Transformer-2020 or RWKV-v1 today you will feel the difference.

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by MAXPOOL

Wed, 24 Jan 2024 15:47:46 GMT

Mamba is a new model architecture based on SSM's. Mamba: Linear-Time Sequence Modeling with Selective State Spaces https://arxiv.org/abs/2312.00752 https://github.com/state-spaces/mamba Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model https://paperswithcode.com/paper/vision-mamba-efficient-visu...

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by lhl

Thu, 11 Jan 2024 17:17:34 GMT

My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with…

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by manojlds

Thu, 04 Jan 2024 01:32:34 GMT

We already have competitors to Transformers https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by CaptainOfCoit

Wed, 20 Dec 2023 16:19:17 GMT

I'm not aware of such a glossary. But I did notice the "References" section in the bottom of the README, which does explain what Mamba is by linking to the original paper: "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by MacsHeadroom

Sat, 16 Dec 2023 09:46:52 GMT

Mamba: Linear-Time Sequence Modeling with Selective State Spaces / https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by squigz

Tue, 12 Dec 2023 03:06:15 GMT

This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the link and read discussion on reddit - if I wanted that sort of discussion, I would browse there, not HN.

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by Koaisu

Tue, 12 Dec 2023 00:04:49 GMT

Here is the paper: https://arxiv.org/abs/2312.00752

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — HN submission

Mon, 04 Dec 2023 20:12:25 GMT

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

130 points; 37 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — mentioned by xenova

Mon, 04 Dec 2023 18:27:02 GMT

Paper: https://arxiv.org/abs/2312.00752 Models: https://huggingface.co/state-spaces

Paper and discussion history. A mention is not an endorsement.

Mamba: Linear-Time Sequence Modeling with Selective State Spaces — HN submission

Mon, 04 Dec 2023 05:03:43 GMT

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.