Prompt Injection as Role Confusion · HackerPapers discussions

Prompt Injection as Role Confusion — mentioned by Giefo6ah

Thu, 27 Aug 2026 20:48:31 GMT

You maybe interested in https://arxiv.org/pdf/2603.12277

Paper and discussion history. A mention is not an endorsement.

Prompt Injection as Role Confusion — mentioned by benob

Tue, 11 Aug 2026 16:03:13 GMT

A natural next step is to use the reasoning traces to jailbreak the stronger models (https://arxiv.org/pdf/2603.12277)

Paper and discussion history. A mention is not an endorsement.

Prompt Injection as Role Confusion — mentioned by simonw

Sun, 09 Aug 2026 00:24:07 GMT

Because nobody has figured out how to make that work 100% reliably yet. The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#s... Then you train your model to take those tokens into account. Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277 > We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds…

Paper and discussion history. A mention is not an endorsement.

Prompt Injection as Role Confusion — mentioned by benob

Fri, 31 Jul 2026 11:05:59 GMT

People will keep hiding reasoning because it allows prompt injection https://arxiv.org/pdf/2603.12277, in addition to facilitating distillation (you don't pay the full cost of RL)

Paper and discussion history. A mention is not an endorsement.

Prompt Injection as Role Confusion — HN submission

Mon, 22 Jun 2026 15:48:55 GMT

Prompt Injection as Role Confusion

https://arxiv.org/abs/2603.12277

235 points; 116 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Prompt Injection as Role Confusion — mentioned by puppystench

Thu, 09 Apr 2026 16:41:45 GMT

I believe you're right, it's an issue of the model misinterpreting things that sound like user message as actual user messages. It's a known phenomenon: https://arxiv.org/abs/2603.12277

Paper and discussion history. A mention is not an endorsement.

Prompt Injection as Role Confusion — mentioned by puppystench

Thu, 09 Apr 2026 15:49:58 GMT

>Several people questioned whether this is actually a harness bug like I assumed, as people have reported similar issues using other interfaces and models, including chatgpt.com. One pattern does seem to be that it happens in the so-called “Dumb Zone” once a conversation starts approaching the limits of the context window. I also don't think this is a harness bug. There's research* showing that models infer the source of text from how it sounds, not the actual role labels the harness would provide. The messages from Claude here sound like user messages ("Please deploy") rather than usual…

Paper and discussion history. A mention is not an endorsement.