An Empirical Study of Mamba-based Language Models · HackerPapers discussions

An Empirical Study of Mamba-based Language Models — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

An Empirical Study of Mamba-based Language Models — mentioned by lhl

Fri, 14 Jun 2024 19:45:36 GMT

You could probably run it as a Q4 (definitely as a Q3) on 4 x A6000 (so on a $25K workstation), although you'd probably also be looking about 3-4 tok/s text generation. I do think that it's a big landmark to have a true GPT4-class model (with some questionable RL though from my initial testing). The best thing about it is that it's almost certainly now the strongest model available for generating synthetic data without any licensing restrictions. Funnily enough, I don't think it's actually the most interesting model that Nvidia released this week. Nvidia also published this paper…

Paper and discussion history. A mention is not an endorsement.

An Empirical Study of Mamba-based Language Models — HN submission

Thu, 13 Jun 2024 17:57:10 GMT

An Empirical Study of Mamba-Based Language Models

43 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.