Efficient Memory Management for Large Language Model Serving with PagedAttention · HackerPapers discussions

Efficient Memory Management for Large Language Model Serving with PagedAttention — mentioned by jbarrow

Mon, 02 Feb 2026 14:18:19 GMT

The whole thing feels AI written, generated from the codebase.* *this is incorrect per the author’s response, my apologies. For instance, it goes into (nano)vLLM internals and doesn’t mention PagedAttention once (one of the core ideas that vLLM is based on)[1]. Also mentions that Part 2 will cover dense vs MoE’s, which is weird because nanovllm hardcodes a dense Qwen3 into the source. Here are better (imo) explainers about how vLLM works: - https://hamzaelshafie.bearblog.dev/paged-attention-from-firs... - https://www.aleksagordic.com/blog/vllm - https://huggingface.co/blog/continuous_batching…

Paper and discussion history. A mention is not an endorsement.

Efficient Memory Management for Large Language Model Serving with PagedAttention — HN submission

Tue, 29 Apr 2025 20:51:44 GMT

Efficient Memory Management for Large Language Model Serving with PagedAttention

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Efficient Memory Management for Large Language Model Serving with PagedAttention — mentioned by jmorgan

Sun, 26 Jan 2025 20:38:28 GMT

Sorry this isn't more obvious. Ideally VRAM usage for the context window (the KV cache) becomes dynamic, starting small and growing with token usage, whereas right now Ollama defaults to a size of 2K which can be overridden at runtime. A great example of this is vLLM's PagedAttention implementation [1] or Microsoft's vAttention [2] which is CUDA-specific (and there are quite a few others). 1M tokens will definitely require a lot of KV cache memory. One way to reduce the memory footprint is to use KV cache quantization, which has recently been added behind a flag [3] and will 1/4 the memory…

Paper and discussion history. A mention is not an endorsement.

Efficient Memory Management for Large Language Model Serving with PagedAttention — mentioned by AaronFriel

Wed, 15 May 2024 17:32:21 GMT

The PagedAttention paper is a good starting point as it represents the first major open source inference engine that had "pretty good" batch performance for transformers. https://arxiv.org/pdf/2309.06180

Paper and discussion history. A mention is not an endorsement.

Efficient Memory Management for Large Language Model Serving with PagedAttention — mentioned by AaronFriel

Thu, 28 Mar 2024 16:49:40 GMT

It indeed is. An attention mechanism's key and value matrices grow linearly with context length. With PagedAttention[1], we could imagine an external service providing context. The hard part is the how, of course. We can't load our entire database in every conversation, and I suspect there are also challenges around training (perhaps addressed via LandmarkAttention[2]) and building a service efficiently retrieve additional key-value matrices. The external service vector database may require tight timings necessary to avoid stalling LLMs. To manage 20-50 tokens/sec, answers must arrive within…

Paper and discussion history. A mention is not an endorsement.

Efficient Memory Management for Large Language Model Serving with PagedAttention — HN submission

Thu, 14 Sep 2023 14:42:03 GMT

Efficient Memory Management for Large Language Model Serving with PagedAttention

102 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.