arXiv:2506.04645cs.LG
Inference economics of language models
Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.
Abstract
We develop a theoretical model that addresses the economic trade-off between cost per token versus serial token generation speed when deploying LLMs for inference at scale. Our model takes into account arithmetic, memory bandwidth, network bandwidth and latency constraints; and optimizes over different parallelism setups and batch sizes to find the ones that optimize serial inference speed at a given cost per token. We use the model to compute Pareto frontiers of serial speed versus cost per token for popular language models.
While the broad point that no, OpenAI and Anthropic are not losing money on inference, is almost certainly correct, the model itself is quite incomplete. If you actually want to know, I recommend Inference economics of language models from Epoch AI, which is probably the best…
sanxiynView on HNParent context