HackerPapers

arXiv:2207.05221cs.CL

Language Models (Mostly) Know What They Know

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, Jared Kaplan

Abstract

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

arXiv comments
23+17 pages; refs added, typos fixed
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 27 comments across 24 threads, from Aug 2022 to Dec 2025. Found in 1 HN submission from Jul 2022 to Jul 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments24 threads

  1. Transformers know more than they can tell: Learning the Collatz sequence 1 comment

    • I believe it's a reference to the paper "Language Models (Mostly) Know What They Know". There's definitely some link but I'd need to give this paper a good read and refresh on the other to see how strong. But I think your final sentence strengthens my suspicion…

      godelskiView on HNParent context

  2. SimpleQA 1 comment

    • >That's not a capability that comes naturally to a probabilistic sampling of tokens. The linked page from OpenAI clearly show the opposite, read the "Using SimpleQA to measure the calibration of large language models" and the paper linked: https://arxiv.org/abs/2207.05221

      GaggiXView on HNParent context

  3. LLMs know more than they show: On the intrinsic representation of hallucinations 1 comment

    • Related: GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets…

      famouswafflesView on HNParent context

  4. GPTs and Hallucination 1 comment

    • Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion: LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance…

      famouswafflesView on HNParent context

  5. AI is growing faster than companies can secure it, warn industry leaders 1 comment

    • That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear. LLMs know a whole lot more about the uncertainty of their predictions than they say. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r…

      famouswafflesView on HNParent context

  6. SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales 1 comment

    • Excellent background on knowledge calibration from Anthropic: https://arxiv.org/abs/2207.05221 "Calibration" in a knowledge context means having estimated_p(correct) ~ p(correct), and it turns out that LLMs are reasonably good at this. Also a core reason why LLM-as-a-judge works…

      xianshouView on HNParent context

  7. Hallucination is inevitable: An innate limitation of large language models 1 comment
  8. Misalignment and Deception by an autonomous stock trading LLM agent 1 comment

    • >What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? There is some indication that models internally understand or at least can distinguish…

      famouswafflesView on HNParent context

  9. I disagree with Geoff Hinton regarding "glorified autocomplete" 1 comment

    • >I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their…

      famouswafflesView on HNParent context

  10. Language models and linguistic theories beyond words 1 comment

    • >An LLM can't tell the difference between fact and fiction, because it can't apply logic. Not true. They can differentiate it just fine. Of course being able to tell the difference and being incentivized to communicate it are 2 different things. GPT-4 logits calibration pre RLHF…

      famouswafflesView on HNParent context

  11. Role play with large language models 1 comment

    • There really is no difference. Either there is utility or there isn't. "Fake" intelligence that produces results is just something that does not make any sense. LLMs may be worse but Humans confidently state something they are uncertain about all the time lol. Maybe not experts…

      famouswafflesView on HNParent context

  12. Rats have an imagination, new research suggests 1 comment

    • >The human model isn’t just trained in language though, it’s trained on the world. It's not trained on "the world". It's trained on a small slice of it, a few senses that are themselves slimmed down and then fabricated at parts. To the bird that can feel and sense…

      famouswafflesView on HNParent context

  13. Ask HN: What is your standard for judging when AGI exists? 2 comments

    • >But the results are indicative of what it is, and they fall short even if your only criteria is usefulness There isn't a testable definition of general intelligence that GPT-4 fails that a chunk of humans also wouldn't. If they fall short then so do people. Guess those people…

      famouswafflesView on HNParent context

    • >More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights. Yeah that's not the reason at all. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated…

      famouswafflesView on HNParent context

  14. The Geometry of Truth: Do LLM's Know True and False 1 comment

    • Related GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their…

      famouswafflesView on HNParent context

  15. "Hallucinating" AIs sound creative, but let's not celebrate being wrong 1 comment

    • Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the…

      famouswafflesView on HNParent context

  16. The Geometry of Truth: Emergent Linear Structure in How LLMs Represent Truth 1 comment

    • Related - GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their…

      famouswafflesView on HNParent context

  17. ChatGPT and other AI tools could disrupt scientific publishing 1 comment

    • >You are unable to. This is just wrong lol. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975…

      famouswafflesView on HNParent context

  18. LLMs can't self-correct in reasoning tasks, DeepMind study finds 2 comments

    • There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores…

      famouswafflesView on HNParent context

    • > Numbers (LLMs still can't do math and the ones that "can" are very brute forced and don't generalize) These seem to generalize. https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2308.00304 > Commutative properties (The (idk why anyone was surprised) "reversal curse"…

      famouswafflesView on HNParent context

  19. How will AI learn next? 1 comment

    • >As a rule, chatbots today have a propensity to confidently make stuff up, or, as some researchers say, “hallucinate.” At the root of these hallucinations is an inability to introspect: the A.I. doesn’t know what it does and doesn’t know. The last bit doesn't seem to be true.…

      famouswafflesView on HNParent context

  20. LLMs confabulate not hallucinate 2 comments

    • Because they don't care to. No seriously that's it. Lots of evidence indicating language models know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting…

      famouswafflesView on HNParent context

    • There's Lots of evidence indicating language models roughly know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language…

      famouswafflesView on HNParent context

  21. The Internal State of a Large Language Model Knows When Its Lying 1 comment

    • Related - GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their…

      famouswafflesView on HNParent context

  22. Avoiding hallucinations in LLM-powered applications 1 comment

    • It has already been done: http://arxiv.org/abs/2207.05221 However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.

      blackbear_View on HNParent context

  23. AlphaFold won’t revolutionise drug discovery 1 comment

    • As someone who works with Transformers and DL in general I decided to chime in: >They can be aware of implicit correlations and constraints that are totally outside what may be recognised in the scientific understanding. Hence AlphaFold is quite capable of predicting a structure…

      rain_iwakuraView on HNParent context

  24. Chinchilla's Wild Implications 1 comment

    • Or maybe hallucinating additional training data: my impression is that AlphaFold was able to get around a lack of sufficient data by training an initial model and having it make a bunch of predictions, then the high-confidence predictions from that set were used to train the…

      brian_cloutierView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.