Parallel Context Windows for Large Language Models — mentioned by Palmik

Sun, 12 Oct 2025 17:07:22 GMT

The observation about the "block-diagonal patterns" in RAG isn't new and has been exploited / explored before: - https://arxiv.org/abs/2410.07590 (literally titled "Block-Attention for Efficient RAG") - https://arxiv.org/abs/2409.15355v3 - https://arxiv.org/abs/2212.10947 The REFRAG paper does not cite any of these.

Paper and discussion history. A mention is not an endorsement.

Parallel Context Windows for Large Language Models — mentioned by 112233

Sat, 28 Jun 2025 11:26:51 GMT

Dropping in late into this discussion, but is there any way to "comfortably" use multiple precomputed kv-caches with current models, in the style of this work: https://arxiv.org/abs/2212.10947 ? Meaning, I pre-parse multiple documents, and the prompt and completion attention sees all of them, but there is no attention between the documents (they are all encoded in the same overlapping positions). This way you can include basically unlimited amount of data in the prompt, paying for it with the perfomance.

Paper and discussion history. A mention is not an endorsement.

Parallel Context Windows for Large Language Models · HackerPapers discussions

Parallel Context Windows for Large Language Models — mentioned by Palmik

Parallel Context Windows for Large Language Models — mentioned by 112233