CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion · HackerPapers discussions

CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — mentioned by wren6991

Fri, 12 Jun 2026 22:41:27 GMT

Prefix caching is already widely deployed by all providers, right? llama.cpp does it. vLLM does it. I'm sure everyone hosting LLMs for a living does it. This paper seems to focus entirely on prefixes (i.e. the prefilled content is rooted at 0). This is... nothing. The referenced CacheBlend paper (https://arxiv.org/pdf/2405.16444) which tries to stitch together multiple independent prefills looks more interesting and is new to me. The problem it's trying to solve is: * KV projections for a given token at a given layer are a function of the residual at that layer, * which is a function of the…

Paper and discussion history. A mention is not an endorsement.

CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — mentioned by tucnak

Sat, 28 Jun 2025 19:53:33 GMT

"Blending," or translating arbitrary substrings to prefixes, is a real curious one, & likely become a prerequisite for running dataset-scale LLM inferences at scale. See https://arxiv.org/abs/2405.16444v3 > To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text’s cross-attention with the preceding texts. Thus, the benefits of…

Paper and discussion history. A mention is not an endorsement.

CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — mentioned by da-x

Sat, 28 Jun 2025 13:16:27 GMT

Yes, there's KV cache 'Blending' see [1]. Future versions of LMCache are aiming to support this. [1] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion- https://arxiv.org/abs/2405.16444

Paper and discussion history. A mention is not an endorsement.