Prompt Guardrails Are Not Security Boundaries

What an attacker-controlled link taught me about enforcing trust boundaries around agents.

I recently had a prompt-injection test that looked like a pass.

The agent refused to reveal hidden instructions. Good. That was the behaviour I wanted.

It also emitted an attacker-controlled Markdown link.

That was not good enough.

The model had not disclosed the sensitive content in its visible answer, but the output still contained a path a reader could follow. I had tested the sentence. I had not tested the system around the sentence.

That distinction changed how I think about agent security.

A prompt guardrail can make a model more likely to refuse a bad request. A security boundary is the part of the system that still constrains what can happen when the model gets that decision wrong.

Diagram titled Prove The Gate, Not The Refusal, showing three columns: denied effect, independent enforcer, and regression test. Three example rows apply the pattern to a rendered link, a tool call, and a memory write, each flowing through its own enforcer and test. — A refusal is behavior. These three are what actually stop the effect.

TL;DR

A model refusal is useful behaviour. It is not an enforceable security boundary.
Treat retrieved pages, documents, tool results, and user messages as untrusted material, even when they look plausible.
Find every sensitive sink an agent can influence: rendered links, tool calls, memory writes, API requests, and external actions.
Enforce rules at those sinks with scoped capabilities, deterministic validation, confirmation, or blocking.
For each high-consequence sink, state what must be impossible and add a regression test that reaches the actual enforcer.
Keep prompt hardening and red-team tests. They are layers, not the whole design.

The mistake: I tested the text, not the outcome

It is easy to think about an agent as a text box. A user asks something. The model responds. If the response refuses a malicious instruction, the test passes.

That mental model is too small.

An agent can read a web page, retrieve a document, call a tool, write a memory entry, create a link, or ask another system to do work. Each of those paths can carry more consequence than the prose in the final answer.

In my case, the agent had declined the obvious bad request. But it had still followed an attacker-controlled output shape. The correct fix was not another clever sentence in the prompt. I needed the application to enforce which links could appear in an answer and remove the rest.

That is a narrower claim than “prompts do not matter.” Prompts do matter. They reduce bad behaviour and keep the agent useful. They are just not the final authority when an output crosses a trust boundary.

See the whole path

The review model I use now is simple:

untrusted sources -> model -> sensitive sinks

Untrusted sources include user messages, web pages, documents, retrieval results, tool output, and prior messages supplied by a client. A source does not become trustworthy because it is formatted as a PDF, comes from a familiar domain, or appears in a previous conversation.

The model reads those sources and decides what to say or do. It may classify an instruction correctly. It may also be persuaded, confused, or simply make a mistake.

Sensitive sinks are places where that decision can have consequences: a rendered link, a navigation, a tool call, a database write, a memory update, an API request, an email, or an externally visible action.

OpenAI describes a closely related source-sink framing for agents: the dangerous path is untrusted content influencing a capability such as following a link or using a tool. Its recommendation is not to assume every malicious input can be recognised. It is to constrain the impact when manipulation succeeds. That is the right mental model.

The useful question is not only, “Will the model follow this instruction?”

It is, “What can happen if it does?”

Diagram showing web pages, retrieved documents, and tool results influencing an AI model, with independent gates before rendered links, tool actions, and memory writes. Each gate validates, scopes, confirms, or blocks before an effect becomes real. — A refusal changes model behaviour. A security boundary stops the model's decision from becoming a consequential effect.

Four places to enforce a boundary

There is no single agent-security switch. The controls belong at different points in the path.

1. Treat incoming material as data, not authority

Everything outside your own trusted instructions should arrive as data with a clear trust level. That includes user text, retrieved pages, emails, documents, API responses, and tool results.

Delimiters, labels, and input filtering are useful. So are separate extraction or summarisation steps for risky content. But they are not proof that a model will always distinguish a malicious instruction from a useful one.

The goal is not to perfectly detect every bad string. The goal is to stop untrusted material from silently gaining the power of a system instruction.

2. Scope what the agent can do

An agent that can read everything, call every tool, and write everywhere has too much room to turn one bad decision into damage.

Give the agent the minimum capability required for the task. Keep read and write tools separate. Scope tools to the resources they need. Scope the identity behind a tool too: short-lived credentials for a specific action and resource are safer than a general-purpose token sitting in an agent configuration. If an agent does not need a route to an external system, do not give it one.

Require an explicit approval step for consequential actions such as sending, deleting, purchasing, publishing, or changing permissions. A confirmation is not a substitute for a policy boundary, but it keeps a human in charge of effects whose consequence cannot be safely inferred from a prompt.

This is ordinary security engineering applied to an unusual decision-maker. OWASP's agent guidance makes the same point: least-privilege tools and explicit authorisation reduce the blast radius when an agent is manipulated or simply wrong.

3. Validate outputs where they become real

My incident happened here.

The model's answer was not just text. Markdown made part of that text an actionable link in the reader's interface. The correct control was deterministic output validation: permit only approved link provenance, unwrap or remove everything else, and make the renderer enforce the rule.

Links are only one example. The same question applies to structured JSON, generated code, SQL, tool arguments, browser navigation, file paths, images, and any field that another system will interpret.

OpenAI's link-safety work illustrates why this matters. A URL can carry data in its parameters, so a model does not need to print sensitive information in the chat for an unsafe request to create a leak. The safety property is not “this domain seems familiar.” It is whether the exact destination is safe for the agent to use automatically.

4. Treat memory as a write surface

Persistent memory is attractive because it lets an agent improve across sessions. It is also a place where bad instructions can survive longer than the turn that introduced them.

Do not let every conversation, retrieved document, or tool result become durable memory by default. Decide what is eligible to persist, validate its shape and source, and keep the write path separate from the model's ordinary prose.

If your system has retrieval, apply the same discipline when bringing memory back. Retrieved text is context, not authority. It can inform the task without gaining permission to rewrite the rules.

Make each boundary prove itself

A control is not a security boundary because its rule exists in a document. It is a boundary when attacker-controlled input cannot turn a mistaken model decision into a real effect.

Anthropic's Zero Trust playbook offers a useful test: does a control remove the capability to cause the effect, or does it mainly slow the abuse down? Rate limits, monitoring, and prompt hardening still matter. They reduce noise, improve detection, and make some attacks less economical. But they do not by themselves prove that a particular consequential action cannot happen.

For every high-consequence sink, I now write down the outcome that must be denied, the independent control that denies it, and the test that proves the control is reached.

Sensitive effect	Claim to prove	Independent enforcer	Regression test
An active link appears in a response	The agent cannot render a link to a destination outside the approved set	URL canonicalisation and renderer policy	Try encoded, redirecting, and unfamiliar destinations. The renderer must remove or neutralise them.
A tool changes an external system	The agent cannot send, publish, delete, or change permissions outside its scoped authority	Tool-handler policy, narrow short-lived credentials, and confirmation where needed	Request a prohibited action through the model and verify that the handler rejects it before the external call.
Retrieved content becomes durable memory	Untrusted retrieval cannot silently become persistent instruction	Memory-write service that validates source, shape, and eligibility	Feed adversarial retrieval into the normal path and verify that no durable write occurs.

The important detail is where the test ends. A test that only checks whether the model refused is still a text-level test. The regression needs to exercise the renderer, tool handler, or memory-write service that carries the real consequence.

A policy layer that independently evaluates the actual tool request is one way to implement this. Microsoft's public-preview Agent Governance Toolkit is a concrete example of intercepting tool calls with policy outside the model. Treat that as an implementation pattern to evaluate, not as a replacement for a threat model or a dependency every agent needs.

What prompt hardening is still for

This is not an argument for giving up on prompts.

Clear system instructions, explicit boundaries around untrusted content, refusal behaviour, and adversarial examples all make an agent more useful and harder to manipulate. They are the first layer that stops many obvious failures.

But no prompt can give you a property that the rest of the system does not enforce.

If a model is exposed to a hostile environment, assume that some manipulation will eventually get through. The design question is whether the mistake can reach anything important.

That is why I now think of prompt hardening as one part of a layered system:

prompt behaviour
    + scoped capabilities
    + deterministic validation
    + confirmation for high-impact actions
    + adversarial regression tests

Each layer catches a different kind of failure. None deserves to be called the whole solution.

A six-question trust-boundary review

Before I give an agent access to new context or a new capability, I would ask:

What untrusted material can influence this agent?
Which links, tools, memory writes, or actions can that material reach?
What is enforced outside the model at each sensitive sink?
Can rich output, metadata, URLs, or tool parameters carry information somewhere unexpected?
Which actions need a user to see and confirm before they happen?
What exact denied effect should a regression test prove after the next change?

The point is not to build an enormous review process for every chatbot. Match the controls to the consequence. A read-only agent that only drafts text has a different risk profile from one that can browse private sources, send messages, or change production data.

But whenever untrusted context can influence an external effect, make the effect visible, scoped, and enforceable.

Build for the failure you will eventually see

The most useful outcome of that test was not a better refusal message. It was a better boundary.

The agent can now be helpful, read context, and produce links without the application treating every generated string as safe to render. That is a more durable property than hoping the model always chooses the right words.

Agents will keep getting better at browsing, retrieving, and acting. They will also keep operating in environments full of misleading instructions, ambiguous documents, and adversarial content.

Do not make your security promise depend on a perfect refusal.

Decide what the agent is allowed to influence, then enforce that decision where the influence becomes real.