Training Deep Neural Networks with 8-bit Floating Point Numbers — mentioned by danuker

Wed, 25 May 2022 12:38:22 GMT

> using a 32 bit integer on a 64 bit platform? Not about integers, but in general: maybe 64-bit is overkill. The trend in ML has been to reduce size: "We have demonstrated DNN training [using nVidia GPUs] with 8-bit floating point numbers (FP8) that achieves 2−4× speedup [compared to FP32 computation] without compromise in accuracy [on most CNN/DNN test accuracy degradation is ~0% stochastically, but max 2% on specific ImageNet runs]." https://arxiv.org/abs/1812.08011

Paper and discussion history. A mention is not an endorsement.

Training Deep Neural Networks with 8-bit Floating Point Numbers · HackerPapers discussions

Training Deep Neural Networks with 8-bit Floating Point Numbers — mentioned by danuker