Mixed Precision Training · HackerPapers discussions

Mixed Precision Training — mentioned by martincsweiss

Thu, 05 Nov 2026 01:41:11 GMT

Thanks for the comment - you're right, of course. I read a bit more into the history of fp32 master copies, like in Mixed Precision Training (Micikevicius et al., ICLR 2018, https://arxiv.org/abs/1710.03740), it says that an fp16 update is lost once the weight is ~2048× larger than it. I've updated the post with a citation to that paper. bf16's shorter mantissa moves that to ~512×, which is significant. The point of the post is narrower: a lot of published fine-tuning code doesn't keep a master copy at all. So what does that do? It gives you a magnitude mask where only the smallest weights…

Paper and discussion history. A mention is not an endorsement.

Mixed Precision Training — mentioned by gdiamos

Wed, 12 Jul 2023 20:01:07 GMT

This initial public alpha release of Lamini LLM fine tuning includes several advanced optimizations: 1. Chinchilla recipe smaller models pretrained on data increases inference speed 2. Instruction fine tuning training on a small high quality set of instructions unlocks the knowledge learned during foundation model training. 3. Latency constrained batching achieves high utilization under load during token generation 4. Containerized SLURM combines fast scheduling of SLURM with LLM containers 5. Mixed precision training uses matrix operations for training There are so many low hanging fruits in…

Paper and discussion history. A mention is not an endorsement.

Mixed Precision Training — mentioned by deepnotderp

Sun, 09 Jun 2019 04:31:27 GMT

It's a popular misconception that neural nets don't really care about precision. 8- bit floats with 16 bit master copies show clear degradation even with careful swamping prevention and stochastic rounding. [0] In practice, you want fp16 with fp32 master copies, and it turns out that if done improperly, even fp32 can cause accuracy losses sometimes [1]. So, you see, it's not actually the case that you can go proceed directly to analog computing without a worry about silly things like correctness, accuracy and precision. [0] https://papers.nips.cc/paper/7994-training-deep-neural-netwo... [1]…

Paper and discussion history. A mention is not an endorsement.

Mixed Precision Training — mentioned by option

Fri, 05 Oct 2018 15:23:49 GMT

Mixed precision training can give you tensor core speedups. Paper: https://arxiv.org/abs/1710.03740 Toolkit which implements it on top of Tensorflow: https://github.com/NVIDIA/OpenSeq2Seq

Paper and discussion history. A mention is not an endorsement.

Mixed Precision Training — mentioned by Smerity

Thu, 26 Oct 2017 09:00:41 GMT

Honestly, I didn't spend enough time delving in to the Python overhead, especially in terms of the framework. Most of it would be an issue of my own causing however rather than the framework's. The original code I wrote was never written with data loading / saving in mind as the source for speed issues so I avoided what would have been premature optimization at the time. Some of the slowdowns now just seem silly and aren't even listed in the per epoch timings: PyTorch doesn't have an asynchronous torch.save(). This means that if you save your model after each epoch, and the model save takes a…

Paper and discussion history. A mention is not an endorsement.

Mixed Precision Training — HN submission

Wed, 11 Oct 2017 17:12:07 GMT

Mixed Precision Training for Deep Learning Models

6 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.