It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization · HackerPapers discussions

It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization — mentioned by psb217

Thu, 08 Jan 2026 10:48:34 GMT

Per your point 4, some current hyped work is pushing hard in this direction [1, 2, 3]. The basic idea is to think of attention as a way of implementing an associative memory. Variants like SDPA or gated linear attention can then be derived as methods for optimizing this memory online such that a particular query will return a particular value. Different attention variants correspond to different ways of defining how the memory produces a value in response to a query, and how we measure how well the produced value matches the desired value. Some of the attention-like ops proposed in this new…

Paper and discussion history. A mention is not an endorsement.

It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization — mentioned by mycall

Fri, 26 Dec 2025 03:11:20 GMT

Have you looked at Titan and MIRAS where they use online/updating associative memory that happens to be read out via next-token prediction? https://research.google/blog/titans-miras-helping-ai-have-lo... https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Much research is going into these directions, but I'm more interested in mind-wandering tangents, involving both attentional control and additional mechanisms (memory retrieval, self-referential processing).

Paper and discussion history. A mention is not an endorsement.

It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization — mentioned by okdood64

Sun, 07 Dec 2025 14:05:36 GMT

From the blog: https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Is there any other company that's openly publishing their research on AI at this level? Google should get a lot of credit for this.

Paper and discussion history. A mention is not an endorsement.