Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting · HackerPapers discussions

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by Betelbuddy

Sat, 17 Oct 2026 13:35:36 GMT

>> Being able to see the CoT Dont assume that is how your model decides... https://arxiv.org/abs/2305.04388 https://arxiv.org/abs/2307.13702 https://arxiv.org/abs/2503.08679

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by recitedropper

Tue, 18 Nov 2025 21:24:56 GMT

"major architectural decisions don't get documented anywhere" "commit messages give no "why"" This is so far outside of common industry practices that I don't think your sentiment generalizes. Or perhaps your expectation of what should go in a single commit message is different from the rest of us... LLMs, especially those with reasoning chains, are notoriously bad at explaining their thought process. This isn't vibes, it is empiricism: https://arxiv.org/abs/2305.04388 If you are genuinely working somewhere where the people around you are worse than LLMs at explaining and documenting their…

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by sabaimran

Wed, 16 Jul 2025 18:51:33 GMT

This is interesting, but I wonder how reliable this type of monitoring is really going to be in the long run. There are fairly strong indications that CoT adherence can be trained out of models, and there's already research showing that they won't always reveal their thought process in certain topics. See: https://arxiv.org/pdf/2305.04388 On a related note, if anyone here is also reading a lot of papers to keep up with AI safety, what tools have been helpful for you? I'm building https://openpaper.ai to help me read papers more effectively without losing accuracy, and looking for more feature…

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by oersted

Tue, 08 Apr 2025 18:16:44 GMT

Recent paper from Anthropic on the fact that the reasoning output does not necessarily reflect what the LLM is "thinking". https://arxiv.org/abs/2305.04388 That being said, your idea is not unreasonable. The way DeepSeek phrased it, it just sounds like implementing such solutions might be a hassle greatly increasing complexity, and they were just focused on making an RL baseline work at scale.

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by grepLeigh

Sun, 09 Feb 2025 22:46:41 GMT

LLMs that use Chain of Thought sequences have been demonstrated to misrepresent their own reasoning [1]. The CoT sequence is another dimension for hallucination. So, I would say that an LLM capable of explaining its reasoning doesn't guarantee that the reasoning is grounded in logic or some absolute ground truth. I do think it's interesting that LLMs demonstrate the same fallibility of low quality human experts (i.e. confident bullshitting), which is the whole point of the OP course. I love the goal of the course: get the audience thinking more critically, both about the output of LLMs and…

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by miven

Tue, 10 Dec 2024 18:08:28 GMT

In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more reliable to let the model run on latents and try to train a separate latent-reading explainer module that has at least some approximation of what we want as an explicit optimization objective. Assuming it actually is or has the potential to be better than CoT, from what I gathered from the paper the current results are mostly just more efficient token-wise. [1]…

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — mentioned by skybrian

Sun, 14 May 2023 21:05:53 GMT

Yes, there's no physical reason it couldn't be fixed. However, I think it's less a matter of computing power, and more one of architecture and training. The inference-time code could be changed to keep a log of which neurons were activated the most when it generated a token. But then, how would we train it to use that log and report it accurately? Training on Internet documents isn't enough. Chain-of-thought reasoning helps as a workaround. However, beware that even then, the justifications aren't a reliable explanation. There's an interesting paper on this: Language Models Don't Always Say…

Paper and discussion history. A mention is not an endorsement.

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — HN submission

Wed, 10 May 2023 03:06:10 GMT

Unfaithful Explanations in Chain-of-Thought Prompting

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.