Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference · HackerPapers discussions

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — HN submission

Thu, 03 Sep 2026 10:27:11 GMT

Understanding, Mitigating Numerical Sources of Nondeterminism in LLM Inference

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — mentioned by shakna

Sun, 30 Aug 2026 14:45:11 GMT

Floating point matrix calculations are non-deterministic. You need to invent new hardware, that doesn't use floating point math, first. [0] [0] https://arxiv.org/html/2506.09501

Paper and discussion history. A mention is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — mentioned by nyrikki

Sat, 14 Mar 2026 04:26:46 GMT

> However, as the name “batch-invariant” suggests, the technique is currently limited to handling variations related only to the batch dimension, making it robust to continuous batching and other batch-size–related changes, but not to other forms of nondeterminism like changing the TP sizes or GPU types. https://arxiv.org/abs/2506.09501

Paper and discussion history. A mention is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — mentioned by rvz

Sat, 07 Feb 2026 02:40:01 GMT

Nope. You have not shown how a large scale collection of neural networks irrespective of their architecture is more deterministic when compared to a 'compiler' and only repeating a known misconception of tweaking the temperature to 0 which does not bring the determinism you claim it brings with LLMs [0] [1] [2], otherwise you would not have this problem in the first place. By even doing that, the result of the outputs are useless anyway. So this really does not help your point at all. So therefore: > You're just using words incorrectly. Deterministic means repeatable. That's it. Predictable…

Paper and discussion history. A mention is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — mentioned by spindump8930

Fri, 11 Jul 2025 19:29:57 GMT

The many sources of stochastic/non-deterministic behavior have been mentioned in other replies but I wanted to point out this paper: https://arxiv.org/abs/2506.09501 which analyzes the issues around GPU non determinism (once sampling and batching related effects are removed). One important take-away is that these issues are more likely in longer generations so reasoning models can suffer more.

Paper and discussion history. A mention is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — mentioned by mzl

Thu, 03 Jul 2025 07:03:40 GMT

LLMs can be run in a mostly deterministic mode (see https://docs.pytorch.org/docs/stable/notes/randomness.html for some info on running PyTorch programs). Varying the deployment type (chip model, number of chips, batch size, ...) can also change the output due to rounding errors. See https://arxiv.org/abs/2506.09501 for some details on that.

Paper and discussion history. A mention is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — HN submission

Wed, 25 Jun 2025 15:34:21 GMT

Give Me FP32 or Give Me Death?

3 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference — mentioned by Der_Einzige

Sun, 22 Jun 2025 15:17:55 GMT

This belief (LLMs are deterministic except for samplers) is very wrong and will get you into hilariously large amounts of trouble for assuming it's true. Also greedy sampling considered harmful: https://arxiv.org/abs/2506.09501 From the abstract: "For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of…

Paper and discussion history. A mention is not an endorsement.