HackerPapers

arXiv:2503.18813cs.CR

Defeating Prompt Injections by Design

Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, Florian Tramèr

Abstract

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving $77\%$ of tasks with provable security (compared to $84\%$ with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.

arXiv comments
Updated version with newer models and link to the code
Journal reference
DOI
Subjects
cs.CRCryptography and Security cs.AIArtificial Intelligence

On Hacker News

Linked by 11 identifiable commenters in 16 comments across 16 threads, from Apr 2025 to Oct 2026. Found in 5 HN submissions from Mar 2025 to Apr 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments16 threads

  1. Is sandboxing sufficient to contain rogue agents? 1 comment

    • >> Perhaps the answer is to have another agent who's goal is not to complete the given task, but to spot cheating or malicious behavior. > That does seem a little like solving the problems in AI by using more of it Yes, and IIRC Google used this as part of a technique against…

      msdzView on HNParent context

  2. What would a serious AI product look like? 1 comment

    • CaMeL, from "Defeating Prompt Injections by Design", https://arxiv.org/abs/2503.18813 See also "CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents", https://arxiv.org/abs/2601.09923v1

      troelsSteeginView on HNParent context

  3. Muse – Meta’s personal AI agent 1 comment

    • Meta AI's David Singleton has a thread about their security prompt injection defenses here: https://twitter.com/dps/status/2097404102468899131 > One threat we’re particularly focused on is prompt injection, and we handle it in layers. The model is trained to recognize and resist…

      simonwView on HNParent context

  4. Launch HN: Prized (YC S26) – Let non-engineer staff build secure internal tools 1 comment

    • We ended up splitting permissions per-tool for data and per-person for connections, for basically this reason. Every tool gets its own db role that can only see its own schema, and integration credentials live in a broker the tool's code never touches. So one tool can't read…

      marinoseliadesView on HNParent context

  5. A GitHub Issue Title Compromised 4k Developer Machines 1 comment

    • Interesting article you’ve linked. I’m not sure I agree, but it was a good read and food for thought in any case. Work is still being done on how to bulletproof input “sanitization”. Research like [1] is what I love to discover, because it’s genuinely promising. If you can…

      msdzView on HNParent context

  6. Don't trust AI agents 1 comment

    • I agree, this is inherently unsafe. The two core security issues for agents, I’d say, are in LLMs not producing a “deterministic” outcome, and prompt injection. Prompt injection is _probably_ solvable if something like [1] ever finds a mainstream implementation and adoption, but…

      msdzView on HNParent context

  7. Implementing a (Vibed) LLM Coding Agent in Prolog 1 comment

    • Great idea to bring the consistency to coding agent with domain specific language (DSL). It reminded me of Google's CaMel work https://arxiv.org/abs/2503.18813 which introduces DSL to LLM prompt to prevent prompt injection by design, and the DeepClause work is like a more…

      phunterlauView on HNParent context

  8. Code Mode: the better way to use MCP 1 comment
  9. From MCP to shell: MCP auth flaws enable RCE in Claude Code, Gemini CLI and more 1 comment

    • The problem is that once you load a tool’s response into context, there’s no telling what the LLM will do. You can escape it all you want, but maybe it contains the right magic words you haven’t thought of. The solution is to not load it into context at all. I’ve seen a proposal…

      brapView on HNParent context

  10. Launch HN: Slashy (YC S25) – AI that connects to apps and does tasks 1 comment

    • That’s a pretty scary answer, to be honest. Regardless, here’s the CaMeL paper. Defeating Prompt Injections by Design (2025): https://arxiv.org/abs/2503.18813 Here’s a paper offering a survey of different mitigation techniques, including CaMeL. Design Patterns for Securing LLM…

      amonksView on HNParent context

  11. GPT-5 1 comment

    • Yup, I have a different definition of expert. I'm not an expert in training models - I'm an expert in applications of those models, and how to explain those applications to other people. AI engineering, not ML engineering, is one way of framing that. I don't write papers (I…

      simonwView on HNParent context

  12. Code execution through email: How I used Claude to hack itself 1 comment

    • It's just YOLO. This problem remains almost entirely unsolved. The closest we've got to what I consider a credible solution is the recent CaMeL paper from DeepMind: https://arxiv.org/abs/2503.18813 - I published some notes on that here…

      simonwView on HNParent context

  13. Supabase MCP can leak your entire SQL database 1 comment

    • I broadly agree that "MCP-level" patches alone won't eliminate prompt-injection risk. Latest research also shows we can make real progress by enforcing security above the MCP layer, exactly as you suggest [1]. DeepMind's CaMeL architecture is a good reference model: it surrounds…

      rexpositoryView on HNParent context

  14. EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot 1 comment
  15. The Policy Puppetry Attack: Novel bypass for major LLMs 1 comment

    • > Is the layoff-based business model really the best use case for AI systems? No, but it plays well in quarterly earnings. So expect a good bit, I'd say. > The flaws are baked into the training data. Not in terms of attack area. In terms of hallucinations... ish. It's more baked…

      groby_bView on HNParent context

  16. Show HN: MCP-Shield – Detect security issues in MCP servers 1 comment

    • I've been following prompt injection for 2.5 years and until last week I hadn't seen any convincing mitigations for it - the proposed solutions were almost all optimistic versions of "if we train a good enough model it won't get tricked any more", which doesn't work. What…

      simonwView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.