Evaluating Quantized Large Language Models · HackerPapers discussions

Evaluating Quantized Large Language Models — mentioned by verdverm

Tue, 14 Jul 2026 21:06:17 GMT

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn. Good evaluation from 2024 https://arxiv.org/pdf/2402.18158 I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.

Paper and discussion history. A mention is not an endorsement.

Evaluating Quantized Large Language Models — mentioned by verdverm

Tue, 14 Jul 2026 20:55:01 GMT

Preliminary analysis via lm-evaluation-harness + vllm model | disk | wikitext | gsm8k (match/error) baseline | 55G | 8.00 | 0.50/0.09 nvfp4-gptq | 27G | 8.25 | 0.47/0.9 nvfp4a16-gptq | 27G | 8.11 | 0.53/0.9 bonsai-4bit | 19G | 16.75 | 0/0 (eval bug?) Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks.…

Paper and discussion history. A mention is not an endorsement.

Evaluating Quantized Large Language Models — HN submission

Fri, 01 Mar 2024 15:16:48 GMT

Evaluating Quantized Large Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.