Fast Matrix Multiplications for Lookup Table-Quantized LLMs · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/2407.10960/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTFast Matrix Multiplications for Lookup Table-Quantized LLMs — mentioned by radichomlhttps://news.ycombinator.com/item?id=41032419urn:hackerpapers:mention:2407.10960:41032419Mon, 22 Jul 2024 09:30:06 GMT<p>Introducing FLUTE, a CUDA kernel for non-uniformly quantized (via a lookup table) LLM Inference. It accelerates QLoRA&#x27;s NormalFloat (NF) out of the box and more. As an application, we extended NF4 and are releasing quantized models for LLaMA-3 (8B/70B) and Gemma-2 (9B/27B). Highlights: - Support arbitrary mapping between (de)quantized tensors via a lookup table (INT4, FP4, NF4, etc). - Up to 2-3x faster than dense GEMM kernels. - 1.3x to 2.6x end-to-end latency improvement (vLLM). - Batch inference, 4/3-bits, and various group sizes. - vLLM integration and 10+ pre-quantized models…</p><p><a href="https://hackerpaper.kaipfister.de/p/2407.10960/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LG