Defeating Prompt Injections by Design · HackerPapers discussions

Defeating Prompt Injections by Design — mentioned by msdz

Thu, 01 Oct 2026 08:03:31 GMT

>> Perhaps the answer is to have another agent who's goal is not to complete the given task, but to spot cheating or malicious behavior. > That does seem a little like solving the problems in AI by using more of it Yes, and IIRC Google used this as part of a technique against prompt injection already [0], back when models were way more susceptible to it. [0] Cf. CaMeL: https://arxiv.org/abs/2503.18813

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by troelsSteegin

Mon, 28 Sep 2026 18:08:52 GMT

CaMeL, from "Defeating Prompt Injections by Design", https://arxiv.org/abs/2503.18813 See also "CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents", https://arxiv.org/abs/2601.09923v1

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by simonw

Wed, 09 Sep 2026 00:16:10 GMT

Meta AI's David Singleton has a thread about their security prompt injection defenses here: https://twitter.com/dps/status/2097404102468899131 > One threat we’re particularly focused on is prompt injection, and we handle it in layers. The model is trained to recognize and resist it. The harness marks anything coming from an untrusted source. Deterministic code checks the result. And an ensemble of classifiers runs where the agent can't reach them. That "deterministic code" bit makes me wonder if they've implemented ideas from the DeepMind CaMeL paper: https://arxiv.org/abs/2503.18813 - my…

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by marinoseliades

Thu, 30 Jul 2026 22:21:55 GMT

We ended up splitting permissions per-tool for data and per-person for connections, for basically this reason. Every tool gets its own db role that can only see its own schema, and integration credentials live in a broker the tool's code never touches. So one tool can't read another tool's data, and nothing the model writes can leak a credential. The ordering thing is harder and currently we don't do it. Per-tool permissions at best ban a combination, they can't say "internet then Slack is ok, the reverse isn't." DeepMind's CaMeL paper (arxiv.org/abs/2503.18813) is the best take I know on…

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by msdz

Fri, 06 Mar 2026 13:08:49 GMT

Interesting article you’ve linked. I’m not sure I agree, but it was a good read and food for thought in any case. Work is still being done on how to bulletproof input “sanitization”. Research like [1] is what I love to discover, because it’s genuinely promising. If you can formally separate out the “decider” from the “parser” unit (in this case, by running two models), together with a small allowlisted set of tool calls, it might just be possible to get around the injection risks. [1] Google DeepMind: Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by msdz

Sat, 28 Feb 2026 16:18:31 GMT

I agree, this is inherently unsafe. The two core security issues for agents, I’d say, are in LLMs not producing a “deterministic” outcome, and prompt injection. Prompt injection is _probably_ solvable if something like [1] ever finds a mainstream implementation and adoption, but agents not being deterministic, as in “do not only what I’ve told you to do, but also how I meant it”, all while assuming perfect context retention, is a waaay bigger issue. If we ever were to have that, software development as a whole is solved outright, too. [1] Google DeepMind: Defeating Prompt Injections by…

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by phunterlau

Thu, 08 Jan 2026 06:46:08 GMT

Great idea to bring the consistency to coding agent with domain specific language (DSL). It reminded me of Google's CaMel work https://arxiv.org/abs/2503.18813 which introduces DSL to LLM prompt to prevent prompt injection by design, and the DeepClause work is like a more general approach on code agent. Great work!

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by dweinus

Sun, 28 Sep 2025 03:39:28 GMT

Nice! Seems very similar to the concept explored by Google (CaMeL) https://arxiv.org/abs/2503.18813

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by brap

Tue, 23 Sep 2025 21:19:33 GMT

The problem is that once you load a tool’s response into context, there’s no telling what the LLM will do. You can escape it all you want, but maybe it contains the right magic words you haven’t thought of. The solution is to not load it into context at all. I’ve seen a proposal for something like this but I can’t find it (I think from Google?). The idea is (if I remember it correctly) to spawn another dedicated (and isolated) LLM that would be in charge of the specific response. The main LLM would ask it questions and the answers would be returned as variables that it may then pass around…

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by amonks

Thu, 04 Sep 2025 23:03:36 GMT

That’s a pretty scary answer, to be honest. Regardless, here’s the CaMeL paper. Defeating Prompt Injections by Design (2025): https://arxiv.org/abs/2503.18813 Here’s a paper offering a survey of different mitigation techniques, including CaMeL. Design Patterns for Securing LLM Agents against Prompt Injections (2025): https://arxiv.org/abs/2506.08837 And here’s a high-level overview of the state of prompt injection from 'simonw (who coined the term), which includes links to summaries of both papers above: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by simonw

Thu, 07 Aug 2025 23:09:35 GMT

Yup, I have a different definition of expert. I'm not an expert in training models - I'm an expert in applications of those models, and how to explain those applications to other people. AI engineering, not ML engineering, is one way of framing that. I don't write papers (I don't have the patience for that), but my work does get cited in papers from time to time. One of my blog posts was the foundation of the work described in the CaMeL paper from DeepMind for example: https://arxiv.org/abs/2503.18813

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by simonw

Thu, 17 Jul 2025 11:03:06 GMT

It's just YOLO. This problem remains almost entirely unsolved. The closest we've got to what I consider a credible solution is the recent CaMeL paper from DeepMind: https://arxiv.org/abs/2503.18813 - I published some notes on that here: https://simonwillison.net/2025/Apr/11/camel/

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by rexpository

Tue, 08 Jul 2025 20:12:19 GMT

I broadly agree that "MCP-level" patches alone won't eliminate prompt-injection risk. Latest research also shows we can make real progress by enforcing security above the MCP layer, exactly as you suggest [1]. DeepMind's CaMeL architecture is a good reference model: it surrounds the LLM with a capability-based "sandbox" that (1) tracks the provenance of every value, and (2) blocks any tool call whose arguments originate from untrusted data, unless an explicit policy grants permission. [1] https://arxiv.org/pdf/2503.18813

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by tough

Fri, 13 Jun 2025 01:24:22 GMT

Google Deepmind published a paper based on this too https://arxiv.org/abs/2503.18813

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — mentioned by groby_b

Tue, 29 Apr 2025 22:46:14 GMT

> Is the layoff-based business model really the best use case for AI systems? No, but it plays well in quarterly earnings. So expect a good bit, I'd say. > The flaws are baked into the training data. Not in terms of attack area. In terms of hallucinations... ish. It's more baked into the structure than the training data. There's increasing amounts of work to ensure well-curated training data. Still, the whole idea of probabilistic selection of tokens more or less guarantees it'll go wrong from time to time. (See e.g. https://arxiv.org/abs/2401.11817) In terms of safety, there's some progress.…

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — HN submission

Mon, 21 Apr 2025 21:19:50 GMT

Defeating Prompt Injections by Design

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Defeating Prompt Injections by Design — HN submission

Sat, 19 Apr 2025 15:22:54 GMT

Defeating Prompt Injections by Design

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Defeating Prompt Injections by Design — HN submission

Sat, 19 Apr 2025 02:20:06 GMT

CaMeL: Defeating Prompt Injections by Design

71 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Defeating Prompt Injections by Design — mentioned by simonw

Tue, 15 Apr 2025 09:50:25 GMT

I've been following prompt injection for 2.5 years and until last week I hadn't seen any convincing mitigations for it - the proposed solutions were almost all optimistic versions of "if we train a good enough model it won't get tricked any more", which doesn't work. What changed is the new CaMeL paper from DeepMind, which notably does not rely on AI models to detect attacks: https://arxiv.org/abs/2503.18813 I wrote my own notes on that paper here: https://simonwillison.net/2025/Apr/11/camel/

Paper and discussion history. A mention is not an endorsement.

Defeating Prompt Injections by Design — HN submission

Fri, 11 Apr 2025 21:37:01 GMT

Defeating Prompt Injections by Design

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Defeating Prompt Injections by Design — HN submission

Wed, 26 Mar 2025 20:31:55 GMT

Defeating Prompt Injections by Design

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.