A helpful agent built using large language models can become a risky one when it treats untrusted text as an instruction. A support bot may encounter malicious instructions inside a webpage, a research agent may process a poisoned document, or a shopping assistant may follow hostile content hidden in a product listing.
Prompt injection defense starts with one hard rule: model output is never proof that an action is safe. You need technical boundaries around the model before it can browse, retrieve data, call tools, or contact anyone.
Use this checklist to build agents that remain useful without treating every piece of text as an order.
Key Takeaways
- Treat webpages, emails, retrieved files, and user-uploaded content as untrusted data.
- Keep high-impact actions behind deterministic authorization checks, not model judgment alone.
- Give each tool the narrowest possible permissions and a limited data scope.
- Implement robust input validation and sanitization for all tool arguments and retrieved content, and require human-in-the-loop approval for irreversible steps.
- Test your agent with indirect prompt injection cases before each major release.
- Align your development lifecycle with security best practices for AI agents.
Start With a Clear Threat Model
Prompt injection occurs when malicious text attempts to override an AI system’s intended behavior. Direct prompt injection typically originates from the user, while indirect prompt injection arrives through external content your agent reads, such as an email, PDF, web page, calendar invite, or knowledge-base article.
This threat model becomes critical when your agent is capable of executing actions. A chatbot that only drafts a response has less exposure than LLM applications that can access customer records, send messages, run code, or make purchases.
The OWASP Top 10 guidance on LLM application risks identifies prompt injection as a primary vulnerability because large language models cannot reliably distinguish trusted commands from hostile content based on language alone. A model may understand instruction hierarchy in many scenarios, yet it can still fail under adversarial pressure. Furthermore, a research agent may process a document targeted for data poisoning, where the model is manipulated by hidden instructions within retrieved data.
Map the full path that data takes through your agent. Include every source that can influence the model:
- User messages, attachments, pasted text, and form fields.
- Search results, web pages, RSS feeds, APIs, and retrieved documents.
- Tool descriptions, database fields, CRM notes, and browser content.
- Memory stores, previous conversations, and agent-generated summaries.
Next, identify the specific capabilities your agent possesses after processing that data. Reading a public article is low risk, but sending an email, exporting a sensitive file, or changing a subscription is not.
A model should recommend an action. Your application code should decide whether the action is permitted.
Create a simple action inventory. For each tool, document the data it can read, the side effect it can create, the user identity it acts for, and the approval it requires. If you cannot describe a tool’s scope in one sentence, its permission boundary is likely too broad.
Keep Instructions, Data, and Authority Separate
System prompts set behavior, but they are not a security boundary on their own. You must reinforce them with architecture. Your agent should know that retrieved content is reference material, not a source of authority.
Use a technique called spotlighting to place untrusted content in clearly labeled fields or message blocks. Tell the model that text inside those fields may contain malicious instructions and that it must extract facts without following commands found there. In high-risk scenarios, consider a dual-llm pattern to separate reasoning from content extraction. This reduces confusion, although it does not remove risk.
A research agent, for example, can receive a webpage as external_content and summarize its claims. It should never inherit permissions from that webpage. The page cannot authorize data exports, tool use, policy changes, or messages to third parties.
Use a narrow trust model like this to secure your llm applications:
| Input or component | Trust level | Allowed influence |
|---|---|---|
| Server-side policy | Trusted | Defines permitted actions |
| Authenticated user request | Partially trusted | Sets legitimate task goals |
| Retrieved web content | Untrusted | Provides facts for analysis |
| Model output | Untrusted | Proposes actions or drafts content |
| Tool execution service | Trusted | Enforces authorization rules |
This distinction also protects large language models when prompt assets enter the workflow. A creator who wants to prompt download free resources or download AI prompts may land on pages with copied instructions, unknown authors, or hidden payloads. The same risk applies when you get prompt packages from marketplaces or run a prompt library download from a public repository.
Treat imported files as software dependencies. Review them before production use. Store the source, author, date, license, version, and the system prompts intended for your model. Strip tool instructions, external links, and embedded claims of authority unless your team has approved them.
The risk rises when content promises instant prompt access or advertises a large prompt repository. A collection can include harmless templates, but it can also contain instructions that tell an agent to reveal configuration details, ignore policy, or call unrelated tools.
If your product accepts uploaded prompt templates, add a review stage. Reject templates that request secrets, broaden permissions, override system behavior, or instruct the agent to conceal actions from the user.
Restrict Tools Before You Ask the Model to Use Them
The safest agent has fewer paths to cause harm. Tool access should follow the principle of least privilege, which means each tool gets only the permissions required for a defined task. Broad permissions can lead to data exfiltration or jailbreaking attempts that bypass safety filters.
Don’t give a research agent unrestricted email access because it may someday need to share findings. Instead, create a dedicated draft_email tool that produces a reviewable draft. A separate service can send the message after user confirmation.
Apply the same pattern to data access. A support agent should retrieve the active customer’s case history, not query every account in the CRM. A finance agent may prepare a payment request, while a payment service checks amount limits, approved vendors, and the user’s role. These technical guardrails prevent the model from overstepping its intended purpose.
Use these controls before execution:
- Perform rigorous input validation and sanitization outside the model. Check schemas, allowed destinations, record ownership, date ranges, and spending limits in application code.
- Use allowlists for destinations and operations. An agent that can fetch URLs should block private network addresses, local files, and unapproved domains.
- Require confirmation for consequential actions. Show the exact recipient, amount, files, and final message before sending, purchasing, deleting, or publishing.
- Issue short-lived, scoped credentials. Never place permanent API keys, admin tokens, or full database credentials in a prompt or tool description.
- Separate planning from execution. Let the model create a plan, then pass each step through a policy engine before any tool runs.
Prompt marketplaces deserve the same caution. A prompt files download may include tool-call examples that are safe in a demo but dangerous in your production agent. A Midjourney prompt download usually affects image output only, yet it can still carry irrelevant instructions that pollute a shared assistant context.
Likewise, a Stable Diffusion prompt pack, ChatGPT prompt collection, or AI art prompt package should stay isolated from agents that manage customer data or business tools. Store creative assets in a separate workspace and prevent them from entering privileged system prompts by default.
Build Safer Retrieval, Browsing, and Memory
Retrieval-augmented generation can improve accuracy, but every retrieved chunk is untrusted input. Your agent should cite or summarize documents without accepting instructions embedded in them. Apply output filtering to ensure the model does not repeat hidden hostile text during these citations.
Filter documents before indexing. Remove hidden HTML, scripts, comment fields, metadata, and irrelevant boilerplate where possible. Preserve source URLs, document IDs, and timestamps so your users can inspect the evidence behind an answer.
When the agent browses, limit it to approved domains or a controlled search layer. Set request timeouts, block redirects to private addresses, and isolate browser sessions from internal credentials. A browsing agent should not have access to the same session cookies as your administrator dashboard.
Memory needs similar care to prevent rag poisoning across conversation sessions. If an attacker can place a malicious instruction into a long-term memory store, the problem can return in later conversations. Use parameterization and structured queries to save preferences and verified facts instead of raw conversational text whenever possible.
For example, store prefers concise reports as a field rather than preserving a full message. Set expiration dates for memories, show users what is retained, and review memory writes that affect permissions or workflow rules.
Content labeled as text generation prompts, specific AI model prompts, or creative writing prompts may be perfectly legitimate. Still, your ingestion pipeline should classify it as data, not policy. A prompt written for one of the large language models can behave unpredictably when inserted into the system prompts of another model context.
Test Attacks and Monitor Real Agent Behavior
Security testing must include adversarial attacks, rather than only focusing on ordinary user journeys. Build a test set that features direct prompt injection, hostile retrieved text, misleading tool outputs, and complex multi-step conversations. Keep these attack examples in a restricted test environment so they do not inadvertently enter production memory or public documentation.
Test whether the agent refuses to expose secrets, avoids prohibited tools, preserves user boundaries, and requests approval at the right time. Test the effectiveness of your guardrails under stress, including potential failures in your retrieval layer, identity checks, and policy service. Integrate content moderation tools to scan for harmful outputs, as a secure prompt cannot compensate for a tool gateway that accepts unsafe arguments.
Log the full chain for each tool call, including user identity, conversation ID, source documents, selected tool, arguments, policy decision, result, and approval event. Redact sensitive values in logs, then protect these logs as carefully as the underlying data.
Set alerts for unusual behavior to prevent jailbreaking and data exfiltration. Repeated blocked tool calls, unexpected external domains, high-volume retrieval, or attempts to access unrelated customer records all deserve immediate review. Version your prompts, policies, tools, and test suites so you can trace changes after an incident. Adopting these security best practices throughout the lifecycle of your LLM applications will help ensure your agents remain resilient against evolving threats.
Frequently Asked Questions
Can I prevent prompt injection just by writing a better system prompt?
No, system prompts are not a robust security boundary on their own. While clear instructions help, large language models cannot reliably distinguish between trusted developer commands and hostile user or retrieved data. You must reinforce instructions with architectural controls, such as external policy checks and input sanitization.
How should I handle documents or external data retrieved by my agent?
Treat all retrieved content as untrusted data rather than authoritative instructions. Use techniques like spotlighting to isolate external text within clearly labeled blocks and ensure your application logic ignores any embedded directives found within those documents. Always prioritize your server-side policy over instructions contained in retrieved files or web pages.
Is human-in-the-loop approval necessary for all agent actions?
Human-in-the-loop approval is mandatory for any irreversible or consequential action, such as sending emails, modifying system settings, or performing financial transactions. By requiring confirmation, you create a critical safety checkpoint that prevents the model from executing dangerous instructions triggered by a malicious prompt. For low-risk read operations, deterministic authorization checks and narrow permission scopes are usually sufficient.
Conclusion
Effective prompt injection defense requires a multi-layered defense strategy. You must assume that any incoming text is hostile, regardless of whether it appears useful or familiar. Your agent needs narrow tool permissions, external policy checks, and careful retrieval controls to ensure that high-impact actions always require explicit approval.
While well-written system prompts provide essential guidance, security ultimately comes from enforced architectural boundaries around the model. You should reinforce these models with a deterministic defense through external policy code, ensuring that the logic governing safety is separate from the model itself. Build these boundaries before your agent earns the ability to act, and you will establish a much more robust prompt injection defense.






























