Prompt Cache: Modular Attention Reuse for Low-Latency Inference — mentioned by xihajun

Wed, 21 Aug 2024 09:26:38 GMT

https://arxiv.org/pdf/2311.04934

Paper and discussion history. A mention is not an endorsement.

Prompt Cache: Modular Attention Reuse for Low-Latency Inference — mentioned by sshumaker

Thu, 16 May 2024 21:30:29 GMT

This is a pretty standard technique if you're running the models yourself. e.g. ChatGPT almost certainly does this. There's even work that is more sophisticated in this domain that allows 'template' style partial caching: https://arxiv.org/abs/2311.04934

Paper and discussion history. A mention is not an endorsement.

Prompt Cache: Modular Attention Reuse for Low-Latency Inference · HackerPapers discussions

Prompt Cache: Modular Attention Reuse for Low-Latency Inference — mentioned by xihajun

Prompt Cache: Modular Attention Reuse for Low-Latency Inference — mentioned by sshumaker