Language Models (Mostly) Know What They Know · HackerPapers discussions

Language Models (Mostly) Know What They Know — mentioned by godelski

Tue, 09 Dec 2025 21:23:34 GMT

I believe it's a reference to the paper "Language Models (Mostly) Know What They Know". There's definitely some link but I'd need to give this paper a good read and refresh on the other to see how strong. But I think your final sentence strengthens my suspicion https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by GaggiX

Thu, 31 Oct 2024 02:22:11 GMT

>That's not a capability that comes naturally to a probabilistic sampling of tokens. The linked page from OpenAI clearly show the opposite, read the "Using SimpleQA to measure the calibration of large language models" and the paper linked: https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Wed, 30 Oct 2024 15:20:37 GMT

Related: GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Tue, 10 Sep 2024 18:53:45 GMT

Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion: LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words). GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Sun, 01 Sep 2024 17:07:31 GMT

That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear. LLMs know a whole lot more about the uncertainty of their predictions than they say. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by xianshou

Tue, 04 Jun 2024 12:13:11 GMT

Excellent background on knowledge calibration from Anthropic: https://arxiv.org/abs/2207.05221 "Calibration" in a knowledge context means having estimated_p(correct) ~ p(correct), and it turns out that LLMs are reasonably good at this. Also a core reason why LLM-as-a-judge works so well: quality evaluation is vastly easier than generation.

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by moyix

Sun, 25 Feb 2024 13:52:00 GMT

Seems to be contradicted by this paper, no? https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Mon, 20 Nov 2023 22:37:23 GMT

>What evidence is there, if any, that LLMs even understand deception as >Deception<? As in, do LLMs understand the concept of Truth, and why other actors might value fidelity to the truth? There is some indication that models internally understand or at least can distinguish truth from falsehood. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Sat, 18 Nov 2023 20:02:19 GMT

>I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception. And yes LLMs can in fact tell truth from fiction. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Thu, 16 Nov 2023 21:23:49 GMT

>An LLM can't tell the difference between fact and fiction, because it can't apply logic. Not true. They can differentiate it just fine. Of course being able to tell the difference and being incentivized to communicate it are 2 different things. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Mon, 13 Nov 2023 22:03:31 GMT

There really is no difference. Either there is utility or there isn't. "Fake" intelligence that produces results is just something that does not make any sense. LLMs may be worse but Humans confidently state something they are uncertain about all the time lol. Maybe not experts in general but then that's still comparing LLMs to a small percentage of humanity. At any rate, Unlike many seem to think, The issue is not in fact the lack of ability to distinguish truth and fact from fiction. Turns out being able to distinguish the two and having the incentive to communicate that are 2 different…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Sun, 12 Nov 2023 19:05:18 GMT

>The human model isn’t just trained in language though, it’s trained on the world. It's not trained on "the world". It's trained on a small slice of it, a few senses that are themselves slimmed down and then fabricated at parts. To the bird that can feel and sense electromagnetic waves intuitively to guide travels, you're not trained on "the world" >This is why they are so prone to hallucination. No it's not. LLMs can distinguish between truth and hallucination. They just don't care to communicate that. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Wed, 08 Nov 2023 18:32:54 GMT

>More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights. Yeah that's not the reason at all. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Wed, 08 Nov 2023 04:37:00 GMT

>But the results are indicative of what it is, and they fall short even if your only criteria is usefulness There isn't a testable definition of general intelligence that GPT-4 fails that a chunk of humans also wouldn't. If they fall short then so do people. Guess those people aren't generally intelligent then according to you. >It can't tell the difference between truth and falsity Yes, it very much can. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Thu, 19 Oct 2023 19:34:31 GMT

Related GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Fri, 13 Oct 2023 19:30:41 GMT

Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the prediction can benefit from all the knowledge/reasoning it has accumulated from training, it'll tap that first. One of the biggest tells of hallucinations is if the model is saying something completely different each generation vs being consistent. What they just don't care about is communicating being out of distribution…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Thu, 12 Oct 2023 23:50:34 GMT

Related - GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Thu, 12 Oct 2023 22:19:54 GMT

>You are unable to. This is just wrong lol. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Mon, 09 Oct 2023 21:00:57 GMT

> Numbers (LLMs still can't do math and the ones that "can" are very brute forced and don't generalize) These seem to generalize. https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2308.00304 > Commutative properties (The (idk why anyone was surprised) "reversal curse": https://arxiv.org/abs/2309.12288) This was a retrieval from training issue not an inference one. and make no mistake, people will fail these sort of answers too (perhaps not to the same extent). Very often someone will learn a fact in a certain why and fail to recall that when asked in a reverse or roundabout way.…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Mon, 09 Oct 2023 19:48:13 GMT

There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Fri, 06 Oct 2023 19:02:05 GMT

>As a rule, chatbots today have a propensity to confidently make stuff up, or, as some researchers say, “hallucinate.” At the root of these hallucinations is an inability to introspect: the A.I. doesn’t know what it does and doesn’t know. The last bit doesn't seem to be true. There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Thu, 05 Oct 2023 07:46:49 GMT

There's Lots of evidence indicating language models roughly know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Thu, 05 Oct 2023 07:34:59 GMT

Because they don't care to. No seriously that's it. Lots of evidence indicating language models know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by famouswaffles

Wed, 27 Sep 2023 12:51:23 GMT

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by blackbear_

Wed, 03 May 2023 11:16:30 GMT

It has already been done: http://arxiv.org/abs/2207.05221 However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by rain_iwakura

Sat, 06 Aug 2022 23:20:59 GMT

As someone who works with Transformers and DL in general I decided to chime in: >They can be aware of implicit correlations and constraints that are totally outside what may be recognised in the scientific understanding. Hence AlphaFold is quite capable of predicting a structure for which there is no previous direct "analogy". Yes, while an expert of their field, I suspect author doesn't have full understanding of what neural networks or ML is capable of, in this case it's not necessary to have seen fully similar molecules but it is sufficient for AlphaFold to have encountered basic blocks of…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — mentioned by brian_cloutier

Wed, 03 Aug 2022 02:37:46 GMT

Or maybe hallucinating additional training data: my impression is that AlphaFold was able to get around a lack of sufficient data by training an initial model and having it make a bunch of predictions, then the high-confidence predictions from that set were used to train the final model. Given that language models seem to mostly know when they're making correct predictions [1] this method might be useful for stretching the available datasets into something larger without falling into the same pitfalls repeating data would give you. And if you squint it kind of looks like daydreaming? When I'm…

Paper and discussion history. A mention is not an endorsement.

Language Models (Mostly) Know What They Know — HN submission

Wed, 13 Jul 2022 17:16:54 GMT

Language Models (Mostly) Know What They Know

43 points; 6 discussion replies on HN. Paper and source history. A submission is not an endorsement.