HackerPapers

arXiv:2210.17323cs.LG

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh

Abstract

Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.

arXiv comments
ICLR 2023
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 15 identifiable commenters in 22 comments across 14 threads, from Mar 2023 to Dec 2023. Found in 1 HN submission from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments14 threads

  1. Do large language models need all those layers? 1 comment

    • I think it's not that LLMs have redundant parameters in general - it's a specific problem with OPT-66B, not anything else. An 2022 paper "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" (http://arxiv.org/abs/2112.11446) has captured it well on page…

      renonceView on HNParent context

  2. Exllamav2: Inference library for running LLMs locally on consumer-class GPUs 1 comment

    • Can anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model? I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a…

      mijoharasView on HNParent context

  3. Nvidia DGX GH200: 100 Terabyte GPU Memory System 1 comment

    • > LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though) From the GPTQ paper https://arxiv.org/abs/2210.17323: "... with negligible accuracy degradation relative to the uncompressed baseline"

      TepixView on HNParent context

  4. The tiny corp raised $5.1M 1 comment

    • When you click on the strip link to preorder the tinybox, it is advertised as a box running LLaMA 65B FP16 for $15000. To be fair, the previous page has a bit more details on the hardware. I can run LLaMA 65B GPTQ4b on my $2300 PC (built from used parts, 128GB RAM, Dual RTX 3090…

      TepixView on HNParent context

  5. Run Llama 13B with a 6GB graphics card 1 comment

    • Training uses gradient descent, so you want to have good precision during that process. But once you have the overall structure of the network, https://arxiv.org/abs/2210.17323 (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy. It…

      sp332View on HNParent context

  6. Matrix Calculus: Calculate derivatives of matrices 1 comment
  7. Llama.cpp 30B runs with only 6GB of RAM now 2 comments

    • For this specific implementation here's info from llama.cpp repo: Perplexity - model options 5.5985 - 13B, q4_0 5.9565 - 7B, f16 6.3001 - 7B, q4_1 6.5949 - 7B, q4_0 6.5995 - 7B, q4_0, --memory_f16 According to this repo[1] difference is about 3% in their implementation with…

      terafoView on HNParent context

    • Define "comprehensive?" There are some benchmarks here: https://www.reddit.com/r/LocalLLaMA/comments/1248183/i_am_cu... and here: https://nolanoorg.substack.com/p/int-4-llama-is-not-enough-i... Check out the original paper on quantization, which has some benchmarks…

      corvecView on HNParent context

  8. Transformer architecture optimized for Apple Silicon 1 comment

    • I'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have…

      garblegarbleView on HNParent context

  9. The genie escapes: Stanford copies the ChatGPT AI for less than $600 2 comments

    • As far as I know, yes. https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the…

      superkuhView on HNParent context

    • > https://arxiv.org/abs/2210.17323 I've read the paper and to be honest I'm not sure what to make of it. Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users. If you look at the tables in the appendix A.4 with some more…

      rnosovView on HNParent context

  10. LLaMa running at 5 tokens/second on a Pixel 6 1 comment

    • They're using GTPQ -- here you go: https://arxiv.org/abs/2210.17323 . The authors benchmarked two families of models over a wide range of numbers of params.

      sottolView on HNParent context

  11. Alpaca: A strong open-source instruction-following model 3 comments

    • Decrease in accuracy is negligible and decreases as model size increases. That is, larger models quantize even better than smaller models. https://arxiv.org/abs/2210.17323

      gamegoblinView on HNParent context

    • WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization. You cannot tell the difference and if you think you do you're wrong, because it barely even registers on any benchmark. (Note: llama.cpp's 4bit is naive, not…

      MacsHeadroomView on HNParent context

    • See: https://arxiv.org/abs/2210.17323 Q: Doesn't 4bit have worsen output performance than 8bit or 16bit? A: GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16.…

      underlinesView on HNParent context

  12. Large language models are having their Stable Diffusion moment 3 comments

    • Apparently if you quantize carefully you can avoid any loss of performance. See https://arxiv.org/pdf/2210.17323.pdf

      sebzim4500View on HNParent context

    • From what I can tell, the quantization being used there is extremely naive. If they were using techniques like GPTQ, presumably it would work much better. https://arxiv.org/pdf/2210.17323.pdf

      sebzim4500View on HNParent context

    • Is 4 bits too small? Or is the quantization method just not sophisticated enough. See https://arxiv.org/pdf/2210.17323.pdf

      sebzim4500View on HNParent context

  13. Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support 3 comments

    • Relevant: Since LLaMA leaked on torrent, it has been converted to Huggingface weights and it has been quantisized to 8bit for less vram requirements. A few days ago it has also been quantisized to 4bit and 3bit is coming. The quantization method they use is from the GPTQ paper…

      underlinesView on HNParent context

    • Yes, but only minimally. Not enough for any human to notice. However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead. References…

      MacsHeadroomView on HNParent context

    • The GPTQ paper https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".

      TepixView on HNParent context

  14. LLaMA Text-Generation-Webui 1 comment

    • text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1] LLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization. LLaMA-30B…

      MacsHeadroomView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.