SqueezeLLM: Dense-and-Sparse Quantization · HackerPapers discussions

SqueezeLLM: Dense-and-Sparse Quantization — mentioned by lhl

Wed, 19 Jul 2023 11:28:14 GMT

ExLlama still uses a bit less VRAM than anything else out there: https://github.com/turboderp/exllama#new-implementation - this is sometimes significant since from my personal experience it can support full context on a quantized llama-33b model on a 24GB GPU that can OOM w/ other inference engines. oobabooga recently did a direct perplexity comparison against various engines/quants: https://oobabooga.github.io/blog/posts/perplexities/ On wikitext, for llama-13b, the perplexity of a q4_K_M GGML on llama.cpp was within 0.3% of the perplexity of a 4-bit 128g desc_act GPTQ on ExLlama, so…

Paper and discussion history. A mention is not an endorsement.

SqueezeLLM: Dense-and-Sparse Quantization — HN submission

Fri, 16 Jun 2023 01:43:39 GMT

SqueezeLLM: Dense-and-Sparse Quantization

5 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

SqueezeLLM: Dense-and-Sparse Quantization — HN submission

Thu, 15 Jun 2023 15:43:48 GMT

SqueezeLLM: Lossless 3-bit quantization with improved performance

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.