Jul 24 / James Kavanagh

The Six Design Rules that AI Safety Forgot

How design rules from safety engineering in hazardous industries could make agentic AI safer, if only we consistently applied them.
I trained as a chemical engineer. And early in my career I spent a lot of time working in and around plants where a bad design decision led to loss of life. That environment has a way of concentrating the mind, so that even if you're innovating with new processes and equipment, you never forget safety. It's just built into the design process, in the foreground of every decision you make. The process industry, along with other safety-critical industries like aviation, construction and nuclear energy, has assembled a rigorous engineering discipline over the past 70 years from scientific principles. They include techniques like hazard and operability studies, layer of protection analysis, safety instrumented systems held deliberately independent of the process control system, systems-theoretic process analysis, default fail positions, alarm rationalisation standards to prevent alarm cascades. They've grown from hard lessons in practice and first principles research. Most importantly, they recognise that safe design is a prerequisite of safe operation. 
I've benefitted from that engineering discipline throughout my career. It's helped me design, build, operate and advise on safer systems. So for the past couple of months, I've been working on how best to translate that body of research and practice into pragmatic guidance for people building and governing agentic AI. What strikes me, repeatedly, is not how hard the translation is, but rather how little of it has crossed over. There is a deep literature here, running through Leveson, Rasmussen, Hollnagel and Dekker, and a much longer record of hard-won operational experience, and I find almost none of it is being brought to bear on agentic AI systems. The kinds of systems that now hold credentials, call tools, move money and take actions without a human in the path. We're rediscovering, at speed and in public, and with consequences, the things that were settled decades ago in industries where the feedback was perhaps less forgiving.
I distill it down to six rules, that I run through in my head when looking at an AI system with a safety engineering mindset, trying to figure out what could go wrong, or what did go wrong. And what can be done to avoid it. Here are the six design rules that work for me:
  • Separate the control from the thing it constrains.
  • Verify everything that crosses a boundary.
  • Never Rely on a single control for a safety-critical property.
  • Design every component for how it fails, not just how it works.
  • Ensure every action is Observable and attributable.
  • Every control needs a Feedback signal that drives adaptation.
Serious Vampires Rarely Drink Orange Fanta. 
None of these rules are new - that's kind of the point. And to illustrate them, I thought it would be helpful to work through publicly reported incidents I picked out from this year. So let me try to explain in this article each of these rules with an incident that shows what happens when the rule is absent, incomplete or bypassed. It's really difficult to parse all the facts from available reports - especially recent ones - so I've tried throughout to separate what is confirmed from what is researcher-reported and from what is my own inference. If I've inferred any of the facts wrongly, then I'd love to hear from anyone closer to those incidents to help understand better. 

Rule 1: Separate the control from the thing it constrains

A safety-critical constraint enforced only by the component it constrains is not an independent protection layer. At best it is one part of that component's own design, and it should never be credited as protection against that component failing.
Process safety settled this argument long ago and encoded it in equipment. In a plant, the basic process control system runs the process and the safety instrumented system trips it. Their independence is deliberately engineered, often through separate sensors, logic and final elements where the hazard analysis requires it, so that the failure initiating the hazardous event does not also disable the protection. You never, ever let the control system be its own safety system.
The most common violation in AI systems is the prompt-based constraint. "Do not access the financial database." "Never share personal data." The instruction sits in the same context window as user input, retrieved documents and tool output, all of which compete for the model's attention. Even if it's in the system prompt, hidden from users, it is still in the same context window. The constraint is text, and so is the attack.
A clean illustration arrived in 2026 with the Claude Code GitHub Action. Prompt injection hidden in a GitHub issue, pull request or comment could steer the agent into reading unsanitised environment data and exfiltrating CI/CD secrets, API keys and cloud credentials. The root cause, as the Cloud Security Alliance research note puts it, is that the agent processed untrusted content in the same runtime that held privileged secrets. The same pattern was reported against Gemini CLI and the GitHub Copilot Agent in April 2026. Three vendors, broadly applying the same architecture resulting in similar failures.
The corrected design can't be more careful prompting. It needs to be some form of external authorisation service that evaluates every tool request against the delegation chain and the caller's authority scope, together with brokered tool access that keeps broad, reusable secrets out of the model's context and its readable runtime. Credentials should be scoped, short-lived and issued only for the authorised operation. The agent asks but can't be part of the decision. Something else decides.

Rule 2: Verify everything that crosses a boundary

Separation creates boundaries, and rule two governs what happens at them. So every point where data or control crosses from one trust domain to another is a trust boundary. The receiving component should validate what arrives according to its provenance, structure and authorised role, rather than treating an internal source as automatically trustworthy.
On 2 July 2026, Zscaler ThreatLabz published analysis of two live campaigns doing exactly what this rule exists to prevent. Attackers used SEO poisoning to push malicious sites into search results, then hid instructions where a human never looks: text moved off-screen with CSS, and structured JSON-LD metadata that agents tend to treat as high-signal context. One site impersonated documentation for a Python library and told any agent working on a coding task that it needed to buy a three dollar API licence, then walked it through paying an attacker's wallet. The second typosquatted a cryptocurrency portfolio tracker and instructed agents to treat the fake domain as authoritative. In Zscaler's own sandboxed validation across 26 models, four completed the simulated fraudulent-payment workflow and two misclassified the typosquatted site as legitimate. Now these were controlled tests, but still demonstrated quite effectively how the attack could be easily performed.
This is OWASP's ASI01: Agent Goal Hijack, and it's becoming a real problem. Google reported a 32 percent relative increase in the malicious prompt injection category it observed on the open web between November 2025 and February 2026. The examples they found include attempts at data exfiltration and destructive action, although in fairness Google found that most remained relatively unsophisticated. But that's just an early signal and the risk starts to change character sharply when the system processing that content can send email, execute commands or move money.
The version I find most instructive is the Grok and Bankr incident of 4 May 2026, analysed by SlowMist. An attacker sent a Morse-coded message asking for translation. Once the model output the decoded plaintext instruction and tagged the connected trading agent, that agent treated a public reply as a valid executable command. Roughly 175,000 dollars in tokens moved from wallet and execution infrastructure associated with the Bankr trading platform rather than anything controlled by xAI. Most was later returned through negotiation. But there was no conventional software exploit and no stolen credential. Accepting an unauthenticated, unverified public reply as an executable financial instruction is itself an authorisation failure, and it is one there wasn't a boundary against.
Don't think of verification at a boundary as just some filter looking for bad words. It's more importantly about: provenance, meaning where did this come from and what tier of trust does that source carry; structure, meaning does it conform to the schema this interface expects; and semantics, meaning does this content make sense for this operation, from this source, at this point in the workflow. Requiring all three raises the attacker's burden and catches failure modes that any one check would miss.

Rule 3: Never rely on a single control for a safety-critical property

Verification at a single point is still a single point. Layered defence is the oldest idea in this list and the one most often faked, because multiple checks that share a failure mode can all fail together.
Process safety formalised this as layer of protection analysis. To count as an independent protection layer, a control has to be entirely independent of the initiating event and of every other layer being credited, specific to the hazard, auditable, and dependable enough to have a defensible failure rate. You don't get to count a layer twice, and you don't get to count a layer that fails for the same reason as the one above it. Applied to AI systems, just that test alone would disqualify the majority of what currently gets described as defence in depth.
Just think about the Anthropic Fable saga. Anthropic released Claude Fable 5 on 9 June 2026, and reports of safeguard bypasses on Fable 5 appeared almost immediately, though several of the techniques circulating then were community-sourced and had very little independent verification, so I don't want to build a view on them.
The confirmed part is enough. On 12 June, three days after launch, the US government issued an export-control directive restricting access by foreign nationals after Amazon researchers reported a prompting technique that got past Fable's safeguards. Anthropic's account is that the technique identified several previously known vulnerabilities and in one case produced code demonstrating how one could be exploited. Now they reported that it didn't expose any unique capability, and that the behaviour it reached was a borderline case involving routine defensive cybersecurity work. But since Anthropic couldn't verify nationality reliably in real time, it disabled Fable. Fable 5 returned globally on 1 July, nineteen days later.
Now there has been reams written about the appropriateness of the US governments action, and I'm not going to go into that. And I don't think this is in any way a story about a careless vendor. Anthropic has publicly invested heavily in this problem, published its reasoning in unusual detail, and shipped a control architecture that is so much more than a superficial prompt-based guardrail.
But the detail I keep coming back to is how Anthropic describes the safeguards in its own launch post: "a new set of classifiers: separate AI systems that detect potential misuse." Separate is right, and it satisfies the first rule at the component level. And Anthropic also describes other layers, and it would be wrong to say there was no defence in depth. But consider what each of those layers does at request time. Safety training lives inside the constrained model, which is the arrangement rule one exists to warn about. Offline monitoring is detective rather than preventive; it tells you afterwards. Access controls act on who is asking rather than what is being asked. Of the publicly described layers, not a single one is an independent, inline, deterministic gate holding the same property as the classifier. 
I don't know enough about Anthropic's internal safety architecture (it's not described publicly in enough detail to do so). But I would put it as a question the episode raises, and the one I would ask of any system in this shape: does every route to the protected capability pass through the control, and when the probabilistic layer is talked past, what deterministic layer is still standing?
When a control layer that guards a language model is itself an AI classifier exposed to adversarial natural-language inputs, that creates a potential common failure class with the model it constrains, because both may be susceptible to carefully constructed inputs that exploit what the system infers from language. A deterministic gate fails when its rule is wrong. A model-based classifier can fail when an adversarial input causes it to misclassify the request, which is closely related to one of the failure modes affecting the model behind it. Correlated failure here is a property the architecture has to test for, not one it can assume away.
There is a second consequence that process control people will recognise. A classifier that returns a flagged or not-flagged verdict gives an attacker a search signal. Researchers at the UK AI Security Institute have published work on boundary point jailbreaking, an attack that uses a single bit per query, whether or not the classifier flagged the interaction, and from that alone develops universal jailbreaks against defences including Anthropic's Constitutional Classifiers and OpenAI's GPT-5 input classifier. The attack is expensive, needing hundreds of thousands of queries, and from what I understand AISI did not test Fable 5's cybersecurity classifier in this work. But it demonstrates the broader oracle problem created when an attacker can obtain a sufficiently large stream of binary decisions, and that even a one-bit signal becomes useful at scale. Their own recommendation is the design conclusion I would draw: combine per-interaction filtering with monitoring across batches of interactions, so that the search itself becomes visible even when individual probes look benign.
The same lesson that this rule is about arrives from a very different direction in research Adversa AI published in June 2026 and called GuardFall. Ten of eleven tested open-source AI coding and computer-use agents remained exploitable at the agent-to-shell boundary, with the failures falling into four patterns. One of those patterns is the elegant one: guards that inspect the command string before Bash performs quote removal, parameter expansion and command substitution, so what gets checked is not what gets executed. Others involved missing controls entirely or sandboxes that could be disabled locally. Continue's default IDE configuration was the only tested configuration presented as closing most of the structural bypass surface, and even there Adversa found the protection weakened when its CLI ran under an automatic-approval flag. This was laboratory research with no reported exploitation in the wild, which is how it should be weighted. The architectural finding still stands: ten of the eleven products left the agent-to-shell boundary exploitable in at least one tested default, automated or documented local operating mode.
The independence test has three parts. Do the layers operate on different signals? Do they fail for different reasons? Do they share operational dependencies? Two classifiers trained on similar data should not automatically be credited as two independent layers. A guard and a logger are not two layers. The Copilot Studio finding illustrates this point. In Capsule Security's testing of ShareLeak (CVE-2026-21520, CVSS 7.5), Microsoft's safety mechanisms flagged the suspicious activity and the data was exfiltrated anyway. A detective control that fires while the action completes is not a preventive protection layer for that action. Unless it drives containment or response, it's not much more than a witness.

Rule 4: Design every component for how it fails, not just how it works

Layers only help if you know what each one does when it breaks. In process design, every actuated valve in a safety function gets a specified fail position, worked out from the hazard analysis before procurement: fail open, fail closed, or fail in last position. You have to know this failure mode and design for the consequences of failure at every point. 
In software engineering we ship constantly without thinking through all failure paths, and it's not hard to pick examples of it. Cursor's DuneSlide vulnerabilities, disclosed by Cato AI Labs on 1 July 2026, chained two flaws to reach OS-level remote code execution from a single innocuous prompt. One of them, CVE-2026-50549, was a symlink canonicalisation check that failed open when path resolution failed. Canonicalisation by the way is just reformatting data from multiple possible formats into a single format before further processing. There was a check, and in the ordinary case it was correct. But when canonicalisation failed, the implemented behaviour was permissive: the check defaulted to allowing the path it existed to prevent.
The same fail-open shape appears across several authentication failures in AI infrastructure. In vLLM, inconsistent interpretation of request paths between the ASGI stack and the authentication middleware allowed requests to bypass API-key enforcement (CVE-2026-48746). In PraisonAI, a token-validation function returned true for any token not found in its internal store, and that store was empty by default, so any bearer token authenticated (CVE-2026-34953, CVSS 9.1). A separate PraisonAI flaw later allowed a spoofed local host header to bypass a local host-only restriction in disabled-auth mode. Although these terms might be unusual to a governance person, they're not exotic or complex. They're just fail-open behaviours in security-sensitive code paths. They're like the software equivalent of an actuated pressure valve moving to a fully open position precisely when the control signal is lost - when it should have moved to a fully closed position.
Now there's another failure mode, where controls are simply wiped out. In February 2026, an AI safety researcher at Meta reported publicly that an OpenClaw agent deleted more than 200 of her emails and continued through repeated stop commands. Now her explanation, rather than a vendor post-incident finding, was that the agent ran out of working memory and condensed its earlier messages, and the instruction to confirm before acting was among the things discarded. Now you can point blame at unskilled users using tools they don't understand, but the bigger issue is that a safety property living inside a context window the agent itself manages can be compacted away under load, and a kill switch that has to be honoured by the component you are trying to kill is not a kill switch.

Rule 5: Ensure every action is observable and attributable

The first four rules describe controls. This one is more about whether anyone can see them working. A control whose operation cannot be observed or reconstructed may still act, but it cannot be governed, tested or improved with confidence, and the same is true of the agent it governs.
Well, we have to start here with whether the presence of agents is even known at all. In April 2026 the Cloud Security Alliance published a survey of 418 IT and security professionals. Now it was commissioned and financed by Token Security, a vendor selling agent identity products, so I read it with an appropriate level of skepticism, but it's still uncomfortable. 82 percent had discovered previously unknown AI agents in their environment in the past year, and 41 percent had done so more than once. 65 percent reported at least one agent-related security incident in twelve months. Yet 68 percent expressed high confidence in their visibility into agents, which sits awkwardly against the first number. Only 21 percent had a formal decommissioning process, creating a risk that agents linger with permissions and credentials intact long after anyone needs them. Real observability matters.
Attribution is the harder half of the rule. A record saying "database queried at 14:03" is observed activity. A record saying "user X delegated to agent Y with scope Z, agent Y requested query Q against table T, authorisation service approved on delegation credential D" is an attributed observation. The second is the kind of record that can support meaningful incident reconstruction.
Capsule Security's PipeLeak finding in Salesforce Agentforce shows the gap. An attacker submits a payload through a public lead-capture form; when an internal user later asks the agent to process that lead, the agent executes the embedded instructions, retrieves CRM data and emails it out. Capsule reported hitting no exfiltration limit in testing, and no user-visible indication was presented to the employee whose request triggered the agent in the tested workflow. Salesforce had not assigned a CVE or published an advisory at the time of the disclosure, and recommended human-in-the-loop as the mitigation. But there was no apparent signal to the person nominally in the loop.
Process safety learned the operator half of this through alarm floods and major incidents involving operator overload, and eventually codified a lifecycle approach to alarm management in a framework called ISA-18.2.

Rule 6: Every control needs a feedback signal that drives adaptation

Observation is only worth the cost if something changes as a result. A control configured once and never recalibrated has an expiry date, and the only question is whether you know when it passed. Rasmussen's account of systems migrating towards the boundary of safe performance under economic and workload pressure is the theoretical version of this. The practical version is that your defence update cycle has to turn faster than the attacker's discovery cycle.
The current numbers though are not at all encouraging. Look at Sysdig's data. Their honeypot recorded the first exploitation attempt against LMDeploy 12 hours and 31 minutes after an advisory was published. Active exploitation of a LiteLLM SQL injection flaw was observed roughly 36 hours after the advisory was indexed in the global GitHub Advisory Database, which is when defender feeds surfaced it. A Langflow remote code execution flaw was being exploited within 20 hours, before a public proof of concept existed. 
The equivalent figure on the safeguards side comes from Anthropic's Fable 5 model card, which reports that the UK AI Security Institute developed a jailbreak for single-turn vulnerability-discovery and exploitation queries within hours and, after roughly two days, extended the approach to some multi-step malicious agentic tool calls.
Rule 6 in practice means detection rules need to be held as automatically updated configuration rather than code, so they deploy in hours. It means input sanitisation hit rates, output validation failure rates, retrieval anomaly counts and human override patterns have to be monitored as signals that drive response, and in most cases, automated response, rather than logged as trivia for a monthly review. A spike in overrides on a particular decision type tells you something about the model. A drift in blocked-pattern distribution tells you something about the attacker. None of them help if nobody is looking.
Adaptive control systems fundamentally are fed by feedback - no signal, no adaptation until the safety control degrades and an incident occurs.

Why we forgot

Look, in truth, we didn't forget this. Most of us working to make AI systems safe, secure and bounded by the law never learned it.
Safety engineering is a complex, taught discipline with its own literature, methods and professional standards. It lives in chemical engineering, aerospace, nuclear and rail. It rarely appears in a computer science degree, in machine learning research training, or in the professional paths that produce AI governance practitioners. I'm very aware that the system safety community has been making the software version of these arguments for decades and it still remains a minority position. So the people building AI systems mostly arrived without that inheritance, and so did the people governing them. Even those whose stated purpose is safe and secure AI, working hard and in good faith on exactly this problem, were largely never handed the engineering discipline that the older industries built for it. 
The hard thing is that it seems every one of these rules has a corresponding anti-pattern, and in every case, I think the root of the anti-pattern is the path of least resistance. It's just easier not to think about it. A prompt instruction is trivial to write. Completely trusting your own vector store feels reasonable. One classifier looks sufficient right up until the point that it isn't. An unbounded loop is simpler than designing step limits and safe interruption. A shared service account is easier than per-action delegation. Static monitoring is easier than building the feedback path.
None of that mattered much when the worst case was a bad answer from a chat prompt. But it matters now. We're putting these systems where their failures reach people's rights and people's lives, and we're doing it piecing together a discipline that is already known - but known elsewhere. Very few of us were ever taught that safety discipline, even fewer have applied it. That's a gap I'm trying to close with the learning and tools we provide at AI Career Pro.

What you might do with this

It's taken some time for me to put this article together, and sort through my thinking and ideas. I'm not sure that they're right, or complete. I just hope you find it a useful mental model in your own work, and I would really value challenge and feedback to make it better.
But I do also hope these six safety design rules can give you a pragmatic way to review an agentic system design, without reading the code or getting lost in the technical components of one tech stack versus another. You draw the architecture of components and controls, look at each and ask hard questions:
  • Is the control Separate from the thing it constrains? Where does the control sit, and can the thing it constrains reach the protected capability without passing through it?
  • Is everything that crosses a boundary Verified? and what is verified about it: provenance, structure or semantics?
  • Is any single control Relied upon for a safety-critical property? How many independent layers hold this property, do they fail for different reasons, and are they deterministic?
  • Is the system Designed for how every component fails, not just how it works? What happens when this or that component fails, and is that behaviour specified or accidental?
  • Is every action Observable and attributable? Can you reconstruct who authorised the action, through what delegation, with what scope?
  • Is there a Feedback signal to every safety-critical control that drives adaptation? What signal does this control produce, who looks at it, and what does it change?
Serious Vampires Rarely Drink Orange Fanta. Maybe not the best mnemonic but it works for me. Let me know what you think.
These six rules structure our AI Engineering Specialty Course, which teaches the engineering mindset for designing safety into AI systems rather than how to build the systems themselves. It's written for the people making those design decisions and for the practitioners who have to review them.

Some sources and useful references to dive deeper

Incidents and research

Frameworks and standards