HackerPapers

arXiv:2503.01781cs.CL

Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models

Meghana Rajeev, Rajkumar Ramamurthy, Prapti Trivedi, Vikas Yadav, Oluwanifemi Bamgbose, Sathwik Tejaswi Madhusudan, James Zou, Nazneen Rajani

Abstract

We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers - short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem's semantics. We propose CatAttack, an automated iterative attack pipeline for generating triggers on a weaker, less expensive proxy model (DeepSeek V3) and successfully transfer them to more advanced reasoning target models like DeepSeek R1 and DeepSeek R1-distilled-Qwen-32B, resulting in greater than 300% increase in the likelihood of the target model generating an incorrect answer. For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models, revealing that even state-of-the-art models remain susceptible to subtle adversarial inputs, raising security and reliability concerns. The CatAttack triggers dataset with model responses is available at https://huggingface.co/datasets/collinear-ai/cat-attack-adversarial-triggers.

arXiv comments
Accepted to CoLM 2025
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 3 threads, from Jul 2025 to Aug 2025. Found in 6 HN submissions from Jul 2025 to Jul 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Training language models to be warm and empathetic makes them less reliable 1 comment

    • Related: https://arxiv.org/abs/2503.01781 > For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong. Also, I think LLMs + pandoc will obliterate junk science in…

      moritzwarhierView on HNParent context

  2. Irrelevant facts about cats added to math problems increase LLM errors by 300% 4 comments

    • When tested against AIs such as DeepSeek V3, Qwen 3, and Phi-4, CatAttack increased the odds of incorrect answers by as much as 700%, depending on the model. And “even when CatAttack does not result in the reasoning model generating an incorrect answer, on average, our method…

      sxvView on HNParent context

    • I am pretty sure this is the paper. https://arxiv.org/abs/2503.01781

      nyrikkiView on HNParent context

    • Seemingly this didn't make frontier models (gpt-o4, gemini-2.5-pro, etc) more likely to give a wrong answer (no stats are reported for failure rates on these models, but slow-down-rate is for similar models), however it does make them think longer sometimes.…

      NoahZunigaView on HNParent context

    • Link to the OG paper: https://arxiv.org/abs/2503.01781

      sean_pedersenView on HNParent context

  3. My open source project was relicensed by a YC company [license updated] 1 comment

    • Get ready to start having some fun in your interviews. Start including things like redirection of focus through general statements, unrelated (and false) trivia, and misleading suggestions in your interview questions. Most of the humans you'd like to hire will ignore those or…

      hoistbypetardView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.