FP8 Quantization: The Power of the Exponent — mentioned by mlgoatherder

Fri, 31 Mar 2023 21:37:30 GMT

I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing. I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressively well but inference is 4-5x slower than native fp16. Oddly I recently ran a fork of the model on the ONNX runtime, I'm convinced that the model performed better than…

Paper and discussion history. A mention is not an endorsement.

FP8 Quantization: The Power of the Exponent · HackerPapers discussions

FP8 Quantization: The Power of the Exponent — mentioned by mlgoatherder