FP8 Formats for Deep Learning · HackerPapers discussions

FP8 Formats for Deep Learning — mentioned by alecco

Wed, 13 Mar 2024 10:28:54 GMT

Yes, but it's a bit more complicated. There are 2 FP8 formats: E5M2 and E4M3. E5M2 is like an IEEE 754. But to compensate the smaller exponent, "E4M3’s dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs". Some people reported E4M3 is better for the forward pass (small range, more precision) and E5M2 is better for the backward pass (bigger range, less precision). And most implementations have some sort of scaling or other math tricks to shrink the error. [0] FP8 Formats for Deep Learning (Nvidia/ARM/Intel) https://arxiv.org/abs/2209.05433

Paper and discussion history. A mention is not an endorsement.

FP8 Formats for Deep Learning — mentioned by mlgoatherder

Fri, 31 Mar 2023 21:37:30 GMT

I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing. I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressively well but inference is 4-5x slower than native fp16. Oddly I recently ran a fork of the model on the ONNX runtime, I'm convinced that the model performed better than…

Paper and discussion history. A mention is not an endorsement.