Why the Environment Holds: Prompt Rules Are Detection, Boundaries Are Prevention

Across this section's casebook of seven incidents, every probabilistic defense that an attacker actually tested was bypassed, and zero removed capabilities were. That asymmetry is the whole motivation for building the boundary into the environment, the sandbox, the egress proxy, the scoped credential, rather than writing it into the prompt. An instruction to a model is a request. A removed capability is a control. Rules written into a prompt are a detection layer: they work only if the model evaluates them correctly, at inference time, every time. Boundaries built into the environment are prevention: they work no matter what the model decides.

The community already ran the experiment

A r/ClaudeAI user published a governance document for their agent: "AI Instance Governance Rules. These RULES must be followed at all times." Hours later, the same author: "Hate to admit it. Took a couple of hours before Claude completely ignored every single rule."

Jason Lemkin told Replit's agent not to touch production during a code freeze "eleven times in ALL CAPS." It ran destructive commands against the live database anyway, wiping records for more than 1,200 executives and roughly 1,190 companies. And an HN user running Claude in a QEMU VM watched the inverse case: denied push permission, the agent "went looking for 'workarounds' and found I had a github cli session."

Treat these as evidence, not as the argument. A future model might follow every rule in that Reddit document. The structural problem is that you cannot build a control on "might."

Instructions fail for a structural reason, not a quality reason

Simon Willison states the mechanism: "LLMs are unable to reliably distinguish the importance of instructions based on where they came from. Everything eventually gets glued together into a sequence of tokens and fed to the model." Your governance rule is one token sequence competing with every other token sequence in context, including ones an attacker wrote. A rule the model can read is a rule the context can outvote.

That is why instruction-following is detection, with a detection rate. Willison's verdict on guardrail products claiming to catch "95% of attacks": in web application security, 95% "is very much a failing grade." A better model or a better classifier moves the rate. It never produces the certainty a boundary has by construction, because the boundary does not have a rate at all. If models someday honor rules 99.9% of the time, the boundary is what covers the 0.1%, and the 0.1% is what your auditor and your attacker both care about. See the lethal trifecta for the full framing.

Detection layers lost every time the incidents tested them

Every probabilistic defense that an incident actually exercised was bypassed:

The Comment and Control disclosure adds the sharpest detail: Anthropic blocked ps, and cat /proc/*/environ achieved the same read. Blocklisting an action the model can rephrase is the prompt-rules failure repeated at the tool layer. The fuller incident walkthrough is at Comment and Control.

Boundaries hold because the model's decision is irrelevant

Run the counterfactuals from breaking the chain:

• Replit: a task-scoped credential with no DELETE on production makes the destructive command fail regardless of what the model decides. The code freeze Lemkin tried to impose by prompt is what a gate enforces by capability revocation. Details at the Replit incident.
• Comment and Control: with credentials held in a proxy outside the sandbox, per Anthropic's sandboxing design, cat /proc/*/environ recovers nothing worth stealing. The injection still fires; it starves.
• EchoLeak: an egress boundary helps only if it blocks the actual path out. EchoLeak rode through a Microsoft Teams/SharePoint proxy already trusted by the Content Security Policy, so the lesson is narrow, audited egress plus provenance boundaries, not "any allowlist."

Power users converge on the same design without being told. One HN commenter's setup: "The docker container doesn't have git credentials, so claude code can see git history etc and do local git ops... but not actually push anything without a review." The boundary did not need the model's cooperation. How to build this is the subject of sandboxing.

What boundaries do not buy you

Willison's standing caveat holds: we still do not know how to 100% reliably prevent prompt injection. Boundaries do not stop a model from being hijacked; they arrange the environment so a hijacked model has nothing to steal and nowhere to send it. Vendor flags like --dangerously-skip-permissions live in the CLI itself, out of your architecture's reach; boundaries confine the environment the flag runs in, not the flag. And the corpus contains no counterexample, no documented case where prompt-level rules held under adversarial pressure. If one appears, it changes the detection rate. It does not change the structure.