Not All LLM Reasoning is Visible in the Chain-of-Thought · HackerPapers discussions

Not All LLM Reasoning is Visible in the Chain-of-Thought — mentioned by dannyw

Mon, 19 Oct 2026 11:45:27 GMT

Yeah, but CoT thinking has been known to be unfaithful for a long time: https://arxiv.org/html/2607.22925v1 Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end. Given global forces wanting models to be smarter, training budget / intelligence…

Paper and discussion history. A mention is not an endorsement.

Not All LLM Reasoning is Visible in the Chain-of-Thought — mentioned by dannyw

Wed, 26 Aug 2026 14:50:35 GMT

You might already know this, but a large part of test-time compute / 'overthinking' is just letting the model do more passes, and refine its activation residuals more. For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 for some starters. Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.

Paper and discussion history. A mention is not an endorsement.