HackerPapers

arXiv:1706.04599cs.LG

On Calibration of Modern Neural Networks

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger

Abstract

Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.

arXiv comments
ICML 2017
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from Apr 2018 to Oct 2026.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers 1 comment

    • Softmax over logits doesn’t give calibrated probabilities either as a rule. I don’t want to comment specifically on Jev but as a rule it’s very hard to get good calibration because it’s somewhat in tension with minimizing training loss for neural networks, e.g. Guo et al (2017)…

      andy99View on HNParent context

  2. Jev in 25 Lines of Python 1 comment

    • In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0]. I ran some tests using GPT-4 to…

      ainchView on HNParent context

  3. Can LLMs learn from a single example? 1 comment

    • Cross-entropy loss can start getting worse due to the model becoming less calibrated, even as the classification accuracy continues to improve. I first heard that here: https://arxiv.org/abs/1706.04599 Is this 'overconfidence' the leading explanation as to why LLMs continue to…

      minihatView on HNParent context

  4. Overconfidence and AI 1 comment

    • Overconfidence has always been a big problem with deep learning. And in fact, better calibration ends up leading to worse overall performance. So ML models are incentivized to be overconfident. To be fair, the same is true of people. See e.g Guo et al, but don't trust their…

      version_fiveView on HNParent context

  5. Smart Folk Often Full of Crap, Study Finds 1 comment

    • The concept of calibration - actually being right at the rate that matches your estimated percent confidence - is important for machine learning. It's important to know how well you can trust the prediction of a ML model (ironically to that you can do something like send low…

      andy99View on HNParent context

  6. Tesla issues strongest statement yet blaming driver for deadly crash 1 comment

    • Have you actually worked on applying neural networks in the real world? Calibrating the probabilities of machine learning algorithms is an old problem. By nature of disciminative algorithms and increasing model capacities, yes training typically pushes outputs to one extreme.…

      EruditassView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.