The case for 4-bit precision: k-bit Inference Scaling Laws · HackerPapers discussions

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by nok22kon

Mon, 29 Jun 2026 23:17:06 GMT

paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bit https://arxiv.org/abs/2212.09720

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by hnfong

Wed, 12 Mar 2025 10:52:54 GMT

Quants at 4 bits are generally considered good, and 8 bits are generally considered overkill unless somehow need to squeeze the last bits of performance (in terms of generation quality) from it. There are papers to that effect though admittedly perhaps specific models might have divergent behavior ( https://arxiv.org/abs/2212.09720 ) All the above is subjective so maybe that’s true for you, but claiming there’s a lack of inference framework for gemma 2 is really off the mark. Obviously ollama supports it. Also llama.cpp. Also mlx. I’ve listed 3 frameworks that support quantized versions of…

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by maratc

Mon, 18 Nov 2024 22:03:55 GMT

> Nvidia B200s ... offer 2-3x the performance of H100s For ML, not for HPC. ML and HPC are two completely different, only loosely related fields. ML tasks are doing great with low precision, 16 and 8 bit precision is fine, arguably good results can be achieved even with 4 bit precision [0][1]. That won't do for HPC tasks, like predicting global weather, computational biology, etc. -- one would need 64 to 128 bit precision for that. Nvidia needs to decide how to divide the billions of transistors on their new silicon. Greatly oversimplifying, they can choose to make one of the following: *…

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by hnfong

Fri, 26 Apr 2024 15:05:49 GMT

There was some speculations in a Reddit thread: https://www.reddit.com/r/LocalLLaMA/comments/1b5uv86/perplex... As far as quantifiable results in terms of perplexity go, q4+ quants are generally considered OK. (eg. https://arxiv.org/abs/2212.09720 )

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by coder543

Mon, 29 Jan 2024 18:38:30 GMT

You don’t stop being andy99 just because you’re a little tired, do you? Being tired makes everyone a little less capable at most things. Sometimes, a lot less capable. In traditional software, the same program compiled for 32-bit and 64-bit architectures won’t be able to handle all of the same inputs, because the 32-bit version is limited by the available address space. It’s still the same program. If we’re not willing to declare that you are a completely separate person when you’re tired, or that 32-bit and 64-bit versions are completely different programs, then I don’t think it’s worth…

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by hnfong

Sat, 27 Jan 2024 15:16:10 GMT

This blog post might shed some light on the matter. If I'm understanding it correctly, it claims there are emergent features on the LLM weights that make it easier to "compress" the floats into smaller bits without losing much precision. https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... Note that 2 bit quantization is generally regarded as too aggressive. Generally 4bits+ achieves a good tradeoff, see eg. https://arxiv.org/abs/2212.09720

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by abhgh

Tue, 07 Nov 2023 09:10:30 GMT

You can make up by learning more parameters, albeit each parameter is of a lower resolution. The tradeoff evidently works favorably till upto 4-bits; I'm basing this off the results reported on the k-bit inference scaling laws paper by Tim Dettmers and Luke Zettlemoyer, see Figure 1 here [1]. [1] https://arxiv.org/abs/2212.09720

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by lhl

Wed, 16 Aug 2023 16:32:10 GMT

Basically, the most aggressive quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684 Tim Dettmers did a bunch of research on this last year as well: https://arxiv.org/abs/2212.09720

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — HN submission

Wed, 26 Jul 2023 14:51:48 GMT

The case for 4-bit precision: k-bit Inference Scaling Laws

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by junrushao1994

Sun, 16 Apr 2023 01:22:16 GMT

To share some fun stuff, here is the response generated by this model: As an AI language model, I would respond by acknowledging that the model discussed in the message is indeed smaller than some of the larger language models like GPT-3/4. However, I would also point out that size is not always a direct indicator of performance, and that the model may still be able to perform well on certain tasks or in certain contexts. Additionally, I would emphasize that the browser compatibility of the model is a significant advantage, as it allows for easy and convenient access to the model without the…

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by corvec

Fri, 31 Mar 2023 21:26:37 GMT

Define "comprehensive?" There are some benchmarks here: https://www.reddit.com/r/LocalLLaMA/comments/1248183/i_am_cu... and here: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i... Check out the original paper on quantization, which has some benchmarks: https://arxiv.org/pdf/2210.17323.pdf and this paper, which also has benchmarks and explains how they determined that 4-bit quantization is optimal compared to 3-bit: https://arxiv.org/pdf/2212.09720.pdf I also think the discussion of that second paper here is interesting, though it doesn't have its own benchmarks…

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by MacsHeadroom

Tue, 14 Mar 2023 01:40:47 GMT

WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization. You cannot tell the difference and if you think you do you're wrong, because it barely even registers on any benchmark. (Note: llama.cpp's 4bit is naive, not GPTQ, and sucks but they are refactoring it to use GPTQ quantization) References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]…

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by MacsHeadroom

Mon, 13 Mar 2023 10:56:40 GMT

Read The Case for 4 Bit Precision. https://arxiv.org/abs/2212.09720 Spoiler: it's the parameter count. As parameter count goes up, but depth matters less. It just so happens that at around 10B+ parameters you can quantize down to 4bit with essentially no downsides. Models are that big now. So there's no need to waste RAM by having unnecessary precision for each parameter.

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by MacsHeadroom

Sun, 12 Mar 2023 00:39:57 GMT

Yes, but only minimally. Not enough for any human to notice. However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead. References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by MacsHeadroom

Fri, 10 Mar 2023 09:29:01 GMT

LLaMA-13B 4bit needs only 18GB of VRAM for GPT-3 175B level text generation. But your point stands. LLaMA-65B 4bit needs 36 GB of VRAM, but far exceeds GPT-3's capabilities and even takes on PaLM 540B. See: https://github.com/oobabooga/text-generation-webui/wiki/LLaM... for 4bit setup instructions See also: The case for 4-bit precision, which shows effectively no output quality reduction for these 4bit quantization methods (and considerable speedup) https://arxiv.org/abs/2212.09720

Paper and discussion history. A mention is not an endorsement.

The case for 4-bit precision: k-bit Inference Scaling Laws — mentioned by woodson

Tue, 03 Jan 2023 19:15:23 GMT

I think quantization (e.g. 4-bit, https://arxiv.org/abs/2212.09720) and sparsity (e.g. SparseGPT, https://arxiv.org/abs/2301.00774) will bring down inference cost. Edit: This isn’t handwaving btw, this is to say some fairly decent solutions are available now.

Paper and discussion history. A mention is not an endorsement.