xLSTM: Extended Long Short-Term Memory · HackerPapers discussions

xLSTM: Extended Long Short-Term Memory — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

xLSTM: Extended Long Short-Term Memory — mentioned by nextos

Fri, 14 Feb 2025 20:21:33 GMT

The xLSTM could become a good alternative to transformers: https://arxiv.org/abs/2405.04517. On very long contexts, such as those arising in DNA models, these models perform really well. There's a big state-space model comeback initiated by the S3-Mamba saga. RWKV, which is a hybrid between classical RNNs and transformers, is also worth mentioning.

Paper and discussion history. A mention is not an endorsement.

xLSTM: Extended Long Short-Term Memory — mentioned by logicchains

Thu, 03 Oct 2024 18:57:27 GMT

The model in the paper isn't a "real" RNN due making it parallelizable, for same the reasons described in https://arxiv.org/abs/2404.08819 , and hence is theoretically less powerful than a "real" RNN (struggles at some classes of problems that RNNs traditionally excel at). On the other hand, https://arxiv.org/abs/2405.04517 contains a "real" RNN component, which demonstrates a significant improvement on the kind of state-tracking problems that transformers struggle with.

Paper and discussion history. A mention is not an endorsement.

xLSTM: Extended Long Short-Term Memory — mentioned by oersted

Wed, 05 Jun 2024 18:29:42 GMT

It does seem to match transformers, but I wouldn't say it meaningfully outperforms them in terms of quality vs parameters. Model: #Params (M), SlimPajama (15B) ppl ↓ - GPT-3: 356M, 14.26 - Llama: 407M, 14.25 - H3: 420M, 18.23 - Mamba: 423M, 13.70 - Hyena: 435M, 17.59 - RWKV-4: 430M, 15.62 - RWKV-5: 456M, 16.53 - RWKV-6: 442M, 17.40 - RetNet: 431M, 16.23 - HGRN: 411M, 21.83 - GLA: 412M, 19.56 - HGRN2: 411M, 16.77 - xLSTM[1:0]: 409M, 13.43 - xLSTM[7:1]: 408M, 13.48 There are more detailed perplexity and task benchmarks in the paper. Overall, all the architectures perform very similarly on every…

Paper and discussion history. A mention is not an endorsement.

xLSTM: Extended Long Short-Term Memory — mentioned by daavidhauser

Thu, 30 May 2024 04:33:37 GMT

xLSTM has a working memory and seems to outperform transformer architectures: https://arxiv.org/abs/2405.04517

Paper and discussion history. A mention is not an endorsement.

xLSTM: Extended Long Short-Term Memory — mentioned by throwaway71271

Sat, 18 May 2024 13:20:11 GMT

360 pages, but also added the xLSTM paper https://arxiv.org/pdf/2405.04517

Paper and discussion history. A mention is not an endorsement.

xLSTM: Extended Long Short-Term Memory — HN submission

Wed, 08 May 2024 05:28:43 GMT

xLSTM: Extended Long Short-Term Memory

197 points; 73 discussion replies on HN. Paper and source history. A submission is not an endorsement.