AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — mentioned by amanrs

Thu, 20 Jul 2023 22:29:49 GMT

The key misconception about many quantization methods is that lower precision = better speed. I believe GPT-Q is not much faster than bf16 from skimming the AWQ paper - https://arxiv.org/pdf/2306.00978.pdf It's 3x faster for a batch size of 1, but that's still over 10x more expensive than gpt-3.5 For larger batch sizes, bf16 costs dip below 3-bit quantized.

Paper and discussion history. A mention is not an endorsement.

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — HN submission

Fri, 02 Jun 2023 15:31:53 GMT

Activation-Aware Weight Quantization for LLM Compression Outperforms GPTQ

9 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · HackerPapers discussions

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — mentioned by amanrs

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — HN submission