Transformers are Multi-State RNNs · HackerPapers discussions

Transformers are Multi-State RNNs — mentioned by in-silico

Thu, 04 Jun 2026 03:19:53 GMT

This is really semantics, but I wouldn't call attending to the KV cache re-reading the context. The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context. I don't know what memory would look like other than an encode-retrieve loop. Relevant: Transformers are Multi-State RNNs - https://arxiv.org/abs/2401.06104

Paper and discussion history. A mention is not an endorsement.

Transformers are Multi-State RNNs — HN submission

Mon, 08 Dec 2025 00:02:54 GMT

Transformers Are Multi-State RNNs

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Transformers are Multi-State RNNs — HN submission

Tue, 16 Jan 2024 20:05:04 GMT

Transformers Are Multi-State RNNs

41 points; 9 discussion replies on HN. Paper and source history. A submission is not an endorsement.