ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization · HackerPapers discussions

ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization — mentioned by cpldcpu

Thu, 08 Oct 2026 22:02:03 GMT

>As for why 4 bit is the limit, the ParetoQ paper has some interesting theories: https://arxiv.org/abs/2502.02631 Its not so clear to me what their real explanation is. Yes, it is possible to redistribute the quantization error, but this only works when the model is not trained to capacity.

Paper and discussion history. A mention is not an endorsement.

ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization — mentioned by nbutton762

Thu, 08 Oct 2026 19:48:07 GMT

First I just want to say, specifically for Post Training Quantization, I absolutely agree with you. As for why 4 bit is the limit, the ParetoQ paper has some interesting theories: https://arxiv.org/abs/2502.02631 That being said, there's a slight misconception about models at lower than 4bpw. There's no fundamental reason why a transformer with low bit weights would be inherently incapable of doing high dimensional function approximation, but training a model at one precision and then quantizing to a lower precision means the training loss is never calculated based on the quantized state.…

Paper and discussion history. A mention is not an endorsement.

ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization — mentioned by kouteiheika

Fri, 30 May 2025 15:20:15 GMT

> The number of quantized bits is a trade off between size and quality. Ideally you should be aiming for a 6-bit or 5-bit model. I've seen some models be unstable at 4-bit (where they will either repeat words or start generating random words). Note that that's a skill issue of whoever quantized the model. In general quantization even as low as 3-bit can be almost loseless when you do quantization-aware finetuning[1] (and apparently you don't even need that many training tokens), but even if you don't want to do any extra training you can be smart as to which parts of the model you're…

Paper and discussion history. A mention is not an endorsement.