MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation · HackerPapers discussions

MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation — mentioned by spmurrayzzz

Fri, 20 Sep 2024 18:48:19 GMT

> they have some long-context "Memory Model" (whose details I've yet to fully understand) Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens". This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models). [1] https://arxiv.org/pdf/2409.05591

Paper and discussion history. A mention is not an endorsement.

MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation — mentioned by diggan

Fri, 20 Sep 2024 18:27:34 GMT

I think leaving just an overview in the repository is fine considering they've released a paper describing it in detail (https://arxiv.org/pdf/2409.05591, linked in the README).

Paper and discussion history. A mention is not an endorsement.

MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation — mentioned by jszymborski

Fri, 20 Sep 2024 16:28:40 GMT

Ok, I think I get it now from scanning the paper and reading Eq. 1 and 2. Normally RAG just sends your query `q` to a information retrieval function which searches a database of documents using full-text search or vector search. Those documents are then passed to a generative model along with your query to give you your final answer. MemoRAG instead immediately passes `q` to a generative model to generate some uninformed response `y`. `y` is then passed to the information retrieval function. Then, just like vanilla RAG, `q` and the retrieved documents are sent to a generative model to give…

Paper and discussion history. A mention is not an endorsement.