Learning to (Learn at Test Time): RNNs with Expressive Hidden States · HackerPapers discussions

Learning to (Learn at Test Time): RNNs with Expressive Hidden States — mentioned by arjvik

Sat, 31 May 2025 19:59:03 GMT

The Titans papers and the Test-Time Training papers (https://arxiv.org/abs/2407.04620) both have the same premise - models should "learn" from their context rather than memorize them. Very promising direction!

Paper and discussion history. A mention is not an endorsement.

Learning to (Learn at Test Time): RNNs with Expressive Hidden States — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

Learning to (Learn at Test Time): RNNs with Expressive Hidden States — mentioned by arjvik

Mon, 11 Nov 2024 18:34:54 GMT

Test-Time Training is incredibly powerful. Most recently, it has been shown that Self-Attention can in fact be viewed through the lens of test-time training, with a kernel-smoother "learning" from context. Simply replacing that with more powerful models than a kernel-smoother result in very capable and scalable models! https://arxiv.org/abs/2407.04620

Paper and discussion history. A mention is not an endorsement.

Learning to (Learn at Test Time): RNNs with Expressive Hidden States — HN submission

Mon, 08 Jul 2024 16:59:46 GMT

Show HN: Learning to (Learn at Test Time)

I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models. We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.” TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context. Our instantiations, TTT-Linear…

13 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Learning to (Learn at Test Time): RNNs with Expressive Hidden States — HN submission

Mon, 08 Jul 2024 14:47:39 GMT

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

10 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.