GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers · HackerPapers discussions

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by renonce

Sat, 16 Dec 2023 04:36:48 GMT

I think it's not that LLMs have redundant parameters in general - it's a specific problem with OPT-66B, not anything else. An 2022 paper "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" (http://arxiv.org/abs/2112.11446) has captured it well on page 103, Appendix G: > The general finding is that whilst compressing models for a particular application has seen success, it is difficult to compress them for the objective of language modelling over a diverse corpus. The appendix G explores various techniques like pruning and distillation but found that neither method was…

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by mijoharas

Wed, 13 Sep 2023 10:04:39 GMT

Can anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model? I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it. [0] https://github.com/turboderp/exllamav2#exl2-quantization [1] https://arxiv.org/abs/2210.17323

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by Tepix

Wed, 31 May 2023 07:42:07 GMT

> LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though) From the GPTQ paper https://arxiv.org/abs/2210.17323: "... with negligible accuracy degradation relative to the uncompressed baseline"

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by Tepix

Thu, 25 May 2023 11:26:30 GMT

When you click on the strip link to preorder the tinybox, it is advertised as a box running LLaMA 65B FP16 for $15000. To be fair, the previous page has a bit more details on the hardware. I can run LLaMA 65B GPTQ4b on my $2300 PC (built from used parts, 128GB RAM, Dual RTX 3090 @ PCIe 4.0x8 + NVLink), and according to the GPTQ paper(§) the quality of the model will not suffer much at all by the quantization. Just saying, open source is squeezing an amazing amount of LLM goodness out of commodity hardware. (§) https://arxiv.org/abs/2210.17323

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by sp332

Sun, 14 May 2023 22:45:39 GMT

Training uses gradient descent, so you want to have good precision during that process. But once you have the overall structure of the network, https://arxiv.org/abs/2210.17323 (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy. It seems you can cut down further for larger models. For the 13B Llama-based ones, going below 5 bits per parameter is noticeably worse, but for 30B models you can do 4 bits. The same group did another paper https://arxiv.org/abs/2301.00774 which shows that in addition to reducing the precision of each parameter, you can…

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — HN submission

Mon, 01 May 2023 16:05:16 GMT

GPTQ Accurate Post-Training Quantization for Generative Pre-Trained Transformers

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by p1esk

Sat, 15 Apr 2023 02:49:47 GMT

Current SOTA post training quantization method is GPTQ: https://arxiv.org/abs/2210.17323

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by corvec

Fri, 31 Mar 2023 21:26:37 GMT

Define "comprehensive?" There are some benchmarks here: https://www.reddit.com/r/LocalLLaMA/comments/1248183/i_am_cu... and here: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i... Check out the original paper on quantization, which has some benchmarks: https://arxiv.org/pdf/2210.17323.pdf and this paper, which also has benchmarks and explains how they determined that 4-bit quantization is optimal compared to 3-bit: https://arxiv.org/pdf/2212.09720.pdf I also think the discussion of that second paper here is interesting, though it doesn't have its own benchmarks…

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by terafo

Fri, 31 Mar 2023 21:16:10 GMT

For this specific implementation here's info from llama.cpp repo: Perplexity - model options 5.5985 - 13B, q4_0 5.9565 - 7B, f16 6.3001 - 7B, q4_1 6.5949 - 7B, q4_0 6.5995 - 7B, q4_0, --memory_f16 According to this repo[1] difference is about 3% in their implementation with right group size. If you'd like to know more, I think you should read GPTQ paper[2]. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa [2] https://arxiv.org/abs/2210.17323

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by garblegarble

Fri, 24 Mar 2023 00:23:35 GMT

I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as they run inference, but they clearly know more than me about the internals), bringing it within reach of phone memory. So, iPhones are not as far off the computational capacity to…

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by rnosov

Mon, 20 Mar 2023 21:41:04 GMT

> https://arxiv.org/abs/2210.17323 I've read the paper and to be honest I'm not sure what to make of it. Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users. If you look at the tables in the appendix A.4 with some more relevant benchmarks you'll sometimes find that straight RTN 4 bit quantisation beats both GPTQ and even full 16 bit original! No explanation of it is given in the paper.

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by superkuh

Mon, 20 Mar 2023 21:27:10 GMT

As far as I know, yes. https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the model. It makes loading it in ram on consumer computers feasible. The number of parameters stays the same.

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by sottol

Wed, 15 Mar 2023 18:01:52 GMT

They're using GTPQ -- here you go: https://arxiv.org/abs/2210.17323 . The authors benchmarked two families of models over a wide range of numbers of params.

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by underlines

Tue, 14 Mar 2023 08:02:08 GMT

See: https://arxiv.org/abs/2210.17323 Q: Doesn't 4bit have worsen output performance than 8bit or 16bit? A: GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16. https://i.imgur.com/xmaNNDd.png https://i.imgur.com/xmaNNDd.png

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by MacsHeadroom

Tue, 14 Mar 2023 01:40:47 GMT

WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization. You cannot tell the difference and if you think you do you're wrong, because it barely even registers on any benchmark. (Note: llama.cpp's 4bit is naive, not GPTQ, and sucks but they are refactoring it to use GPTQ quantization) References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]…

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by gamegoblin

Mon, 13 Mar 2023 19:16:33 GMT

Decrease in accuracy is negligible and decreases as model size increases. That is, larger models quantize even better than smaller models. https://arxiv.org/abs/2210.17323

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by Tepix

Mon, 13 Mar 2023 11:14:07 GMT

The GPTQ paper https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by sebzim4500

Sun, 12 Mar 2023 17:54:37 GMT

Is 4 bits too small? Or is the quantization method just not sophisticated enough. See https://arxiv.org/pdf/2210.17323.pdf

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by sebzim4500

Sun, 12 Mar 2023 17:53:53 GMT

From what I can tell, the quantization being used there is extremely naive. If they were using techniques like GPTQ, presumably it would work much better. https://arxiv.org/pdf/2210.17323.pdf

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by sebzim4500

Sun, 12 Mar 2023 17:38:06 GMT

Apparently if you quantize carefully you can avoid any loss of performance. See https://arxiv.org/pdf/2210.17323.pdf

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by MacsHeadroom

Sun, 12 Mar 2023 00:39:57 GMT

Yes, but only minimally. Not enough for any human to notice. However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead. References: https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by underlines

Sat, 11 Mar 2023 10:20:21 GMT

Relevant: Since LLaMA leaked on torrent, it has been converted to Huggingface weights and it has been quantisized to 8bit for less vram requirements. A few days ago it has also been quantisized to 4bit and 3bit is coming. The quantization method they use is from the GPTQ paper ( https://arxiv.org/abs/2210.17323 ) which leads to almost no quality degradation compared to the 16bit weights. 4 bit weights: Model, weight size, vram req. LLaMA-7B, 3.5GB, 6GB LLaMA-13B, 6.5GB, 10GB LLaMA-30B, 15.8GB, 20GB LLaMA-65B, 31.2GB, 40GB Here is a good overall guide for Linux and Windows…

Paper and discussion history. A mention is not an endorsement.

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — mentioned by MacsHeadroom

Fri, 10 Mar 2023 21:49:54 GMT

text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1] LLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization. LLaMA-30B fits on a 24GB* consumer video card with no output performance loss, beating GPT-3 175B. Multi-GPU support[2] means LLaMA-65B, rivaling PaLM-540B, runs on 2x3090. *Further improvements in active development will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention). [0]…

Paper and discussion history. A mention is not an endorsement.