Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback · HackerPapers discussions

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by dcx

Mon, 19 Jan 2026 04:38:48 GMT

Hello! I'm one of the main authors of the paper. Thanks for engaging with our work so thoughtfully – that's a very clear and valid question. We didn't get around to addressing this within the paper itself – 80 pages is a lot, and deadlines, etc. But I have unpublished experiments that show that in a reasonably broad setting I'm doing some work in, verbalized probabilities are restoring a distribution that looks almost identical to the base distribution. It is not possible to demonstrate this on frontier models, since their public models are already mode-collapsed, and they don't share the…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Wed, 30 Oct 2024 15:20:37 GMT

Related: GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-... Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Tue, 10 Sep 2024 18:53:45 GMT

Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion: LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words). GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Sun, 01 Sep 2024 17:07:31 GMT

That's not true. There's been several papers probing this with different methodologies and the conclusion is pretty clear. LLMs know a whole lot more about the uncertainty of their predictions than they say. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824 The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734 LLMs Know More…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by cosmojg

Thu, 07 Dec 2023 05:33:29 GMT

What was the point of moving away from the base model? I can't stop asking this question. Conversational formatting is achievable with careful prompting and a bit of good old-fashioned heuristic post-processing, and it was easier to achieve consistent results before RLHF took off. Now we still have to do a bunch of prompt hacking to get the results we want[1], but it's more complicated and the performance of the model has degraded significantly[2]. All the cargo culting toward agentic chatbots and away from language prediction engines might please the marketing and investor relations…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Sat, 18 Nov 2023 20:02:19 GMT

>I know my world model is fundamentally incomplete. Even more foundationally, I know that there is a world, and when my world model and the world disagree, the world wins. Yeah this isn't really true. There's not how humans work. For a variety of reasons, Plenty stick with their incorrect model despite the world indicating otherwise. In fact, this seems to be normal enough human behaviour. Everyone does it, for something or the other. You are no exception. And yes LLMs can in fact tell truth from fiction. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 16 Nov 2023 21:23:49 GMT

>An LLM can't tell the difference between fact and fiction, because it can't apply logic. Not true. They can differentiate it just fine. Of course being able to tell the difference and being incentivized to communicate it are 2 different things. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Mon, 13 Nov 2023 22:03:31 GMT

There really is no difference. Either there is utility or there isn't. "Fake" intelligence that produces results is just something that does not make any sense. LLMs may be worse but Humans confidently state something they are uncertain about all the time lol. Maybe not experts in general but then that's still comparing LLMs to a small percentage of humanity. At any rate, Unlike many seem to think, The issue is not in fact the lack of ability to distinguish truth and fact from fiction. Turns out being able to distinguish the two and having the incentive to communicate that are 2 different…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Sun, 12 Nov 2023 19:05:18 GMT

>The human model isn’t just trained in language though, it’s trained on the world. It's not trained on "the world". It's trained on a small slice of it, a few senses that are themselves slimmed down and then fabricated at parts. To the bird that can feel and sense electromagnetic waves intuitively to guide travels, you're not trained on "the world" >This is why they are so prone to hallucination. No it's not. LLMs can distinguish between truth and hallucination. They just don't care to communicate that. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Wed, 08 Nov 2023 18:32:54 GMT

>More importantly, this is because it doesn't understand the meaning of what it generates, just juggles weights. Yeah that's not the reason at all. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 The Geometry of Truth: Emergent Linear Structure in…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Wed, 08 Nov 2023 04:37:00 GMT

>But the results are indicative of what it is, and they fall short even if your only criteria is usefulness There isn't a testable definition of general intelligence that GPT-4 fails that a chunk of humans also wouldn't. If they fall short then so do people. Guess those people aren't generally intelligent then according to you. >It can't tell the difference between truth and falsity Yes, it very much can. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 19 Oct 2023 19:34:31 GMT

Related GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Fri, 13 Oct 2023 19:30:41 GMT

Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data). This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the prediction can benefit from all the knowledge/reasoning it has accumulated from training, it'll tap that first. One of the biggest tells of hallucinations is if the model is saying something completely different each generation vs being consistent. What they just don't care about is communicating being out of distribution…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 12 Oct 2023 23:50:34 GMT

Related - GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 12 Oct 2023 22:19:54 GMT

>You are unable to. This is just wrong lol. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — HN submission

Tue, 10 Oct 2023 23:15:52 GMT

Just Ask for Calibration: Eliciting Calibrated Confidence Scores from LLMs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Mon, 09 Oct 2023 21:00:57 GMT

> Numbers (LLMs still can't do math and the ones that "can" are very brute forced and don't generalize) These seem to generalize. https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2308.00304 > Commutative properties (The (idk why anyone was surprised) "reversal curse": https://arxiv.org/abs/2309.12288) This was a retrieval from training issue not an inference one. and make no mistake, people will fail these sort of answers too (perhaps not to the same extent). Very often someone will learn a fact in a certain why and fail to recall that when asked in a reverse or roundabout way.…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Mon, 09 Oct 2023 19:48:13 GMT

There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Fri, 06 Oct 2023 19:02:05 GMT

>As a rule, chatbots today have a propensity to confidently make stuff up, or, as some researchers say, “hallucinate.” At the root of these hallucinations is an inability to introspect: the A.I. doesn’t know what it does and doesn’t know. The last bit doesn't seem to be true. There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 05 Oct 2023 17:41:07 GMT

No that's just a training process that goes, "people like this sort of responses better, try to emulate it" by feeding specific input into the training process. >So how do we get LLMs to admit when they don't know something rather than confabulate? The prompts here demonstrably help https://arxiv.org/abs/2305.14975 But otherwise nobody is certain how to do it. The simplest solution is just making them better. Hallucination is something that happens when knowledge/memory fail. By making them more competent, they hallucinate less. So even if they never care about communicating this, you…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 05 Oct 2023 07:46:49 GMT

There's Lots of evidence indicating language models roughly know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Thu, 05 Oct 2023 07:34:59 GMT

Because they don't care to. No seriously that's it. Lots of evidence indicating language models know when they hallucinate. GPT-4 logits calibration pre RLHF - [https://imgur.com/a/3gYel9r](https://imgur.com/a/3gYel9r) Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - [https://arxiv.org/abs/2305.14975](https://arxiv.org/abs/2305... Teaching Models to Express Their Uncertainty in Words - [https://arxiv.org/abs/2205.14334](https://arxiv.org/abs/2205... Language Models (Mostly) Know What They Know…

Paper and discussion history. A mention is not an endorsement.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — mentioned by famouswaffles

Wed, 27 Sep 2023 12:51:23 GMT

Paper and discussion history. A mention is not an endorsement.