Landmark Attention: Random-Access Infinite Context Length for Transformers · HackerPapers discussions

Landmark Attention: Random-Access Infinite Context Length for Transformers — mentioned by AaronFriel

Thu, 28 Mar 2024 16:49:40 GMT

It indeed is. An attention mechanism's key and value matrices grow linearly with context length. With PagedAttention[1], we could imagine an external service providing context. The hard part is the how, of course. We can't load our entire database in every conversation, and I suspect there are also challenges around training (perhaps addressed via LandmarkAttention[2]) and building a service efficiently retrieve additional key-value matrices. The external service vector database may require tight timings necessary to avoid stalling LLMs. To manage 20-50 tokens/sec, answers must arrive within…

Paper and discussion history. A mention is not an endorsement.

Landmark Attention: Random-Access Infinite Context Length for Transformers — mentioned by Tostino

Thu, 01 Jun 2023 15:08:13 GMT

Completely agree here. And the fact of the matter is, long term memory is actively being worked on. You can already approximate reasonably with a Vector DB and embeddings, feeding the relevant results into the context. As is increasing the "working memory" (context size) in a more efficient manner: https://arxiv.org/abs/2305.16300 This isn't even getting into using things like Tree of Thought techniques to improve reasoning abilities and limit the search space for each portion of the problem to a reasonable size.

Paper and discussion history. A mention is not an endorsement.

Landmark Attention: Random-Access Infinite Context Length for Transformers — HN submission

Sat, 27 May 2023 22:09:44 GMT

Landmark Attention: Random-Access Infinite Context Length for Transformers

22 points; 4 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Landmark Attention: Random-Access Infinite Context Length for Transformers — HN submission

Sat, 27 May 2023 11:39:12 GMT

Random-Access Infinite Context Length for Transformers

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.