HackerPapers

arXiv:2507.11473cs.AI

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Mądry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, Vlad Mikulik

Abstract

AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.

arXiv comments
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Jan 2026 to Oct 2026. Found in 1 HN submission from Jul 2025 to Jul 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Hybrid Reasoning Agents: LLM agents that think in latent space, act through text 1 comment

    • > There is genuine research into latent space monitoring and interpretation That's great but the sentiment that CoT isn't useful for monitoring doesn't seem to be shared by mech interp researchers like Nanda. Also: https://arxiv.org/abs/2507.11473

      rarpaView on HNParent context

  2. Claude Opus 4.7 1 comment

    • There’s also a paper [0] from many well known researchers that serves as a kind of informal agreement not to make the CoT unmonitorable via RL or neuralese. I also don’t think Anthropic researchers would break this “contract”. [0] https://arxiv.org/abs/2507.11473

      clbrmbrView on HNParent context

  3. System Card: Claude Mythos Preview [pdf] 1 comment

    • I agree that they called many things remarkably well! That doesn't change the fact that AI 2027 is not a thing which happened, so it isn't valid to point out "this killed us in AI 2027." There are many reasons to want to preserve CoT monitorability. Instead of AI 2027, I'd point…

      Turn_TroutView on HNParent context

  4. Scaling Latent Reasoning via Looped Language Models 1 comment

    • Yes. Coconut has the same issue. See also: a joint statement by researchers from several labs about CoT monitorability: https://arxiv.org/abs/2507.11473

      the8472View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.