OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models — HN submission

Mon, 08 Apr 2024 16:07:18 GMT

Show HN: The fastest way to run Mixtral 8x7B on Apple Silicon Macs

I’d originally launched my app: Private LLM[1][2] on HN around 10 months ago, with a single RedPajama Chat 3B model. The app has come a long way since then. About a month ago, I added support for 4-bit OmniQuant quantized Mixtral 8x7B Instruct model, and it seems to outperform Q4 models at inference speed and Q8 models at text generation quality, while consuming only about 24GB of RAM[3] at 8k context length. The trick is: a) to use a better quantization algorithm and b) to use unquantized embeddings and the MoE gates (the overhead is quite small). Other notable features include many more…

18 points; 22 discussion replies on HN. Paper and source history. A submission is not an endorsement.

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models · HackerPapers discussions

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models — HN submission