May 28 / James Kavanagh

How I Triage AI Governance Problems

A practical approach to identifying the right problems to work on
The Danish safety researcher Jens Rasmussen spent much of his career studying how complex systems fail. Unlike many others in his field, he didn't may much attention to how they fail in theory, in the neat diagrams of fault trees and risk matrices, but how they fail in practice. Probably why I like his writing so much. And one of his most important insights, developed through decades of studying real accidents in industries from nuclear power to healthcare, was this: systems don't usually fail because someone does something obviously dangerous. They fail because lots of people make small, individually reasonable decisions that each move the system a little closer to a boundary it was never meant to cross.
He called this migration. Let me give you an example. Let's say a research team builds a model to explore a clinical question. It works well. A clinician sees the outputs and starts using them informally to guide a decision. Someone integrates the scores into a dashboard because it saves time. Months later, the model is shaping which patients get prioritised for care, and nobody ever made a conscious decision to deploy it in that role. Each step made sense. Nobody did anything wrong. And yet the system has drifted from a research prototype to an operational decision-maker without a risk assessment, without validation, without monitoring, and without anyone noticing. 
Drift to danger: the systemic migration of organizational behavior toward accident under the influence of pressure toward cost-effectiveness in an aggressive, competing environment. - Rasmussen (1)
Rasmussen's point was that this kind of drift is not unusual. It's the normal behaviour of organisations under pressure to be efficient, to move fast, to get things done. People don't drift toward danger because they're negligent. They drift because the boundary between "acceptable" and "not acceptable" is invisible. There's no alarm. No warning sign. No bright line that tells you today is the day this research tool became a clinical system. That drift is gradual, and each step looks fine from where you're standing.
Rasmussen is my guide when I talk to an organisation that's just starting to take AI governance seriously. If you're just starting or progressing through the early stages of your career in AI governance, let me illustrate how it commonly plays out.
Someone asks you to review one AI system (maybe that's even the basis of you being hired). Maybe deployment of that system is blocked, it triggered a complaint, or a supplier demands governance artifacts, or a regulator asked questions, or a board member read something alarming. You arrive, and within hours you discover that the system you were hired to review is one of a dozen AI systems operating across the organisation. Some were procured. Some were built internally. Some drifted from prototype into operations exactly the way Rasmussen described: one reasonable step at a time, nobody noticing the accumulation. Some are shadow AI: staff using consumer tools on personal devices, pumping sensitive data into services nobody sanctioned.
And most of those systems have no governance at all. No risk assessment, no monitoring, no incident process, no documented accountability. The system that led to you being there, the one that received all the attention, is possibly the lowest risk of the lot. The governance attention was inversely proportional to the real risk. 
It's a bit like everything is inverted - the systems that warrant the most attention get the least, and the new shiny, agentic chatbot is somehow what everyone is focused on. Rasmussen explains that. Organisations govern what they can see. The systems that arrive through formal procurement channels, that get flagged by clinicians or engineers or clients, and those systems get attention. The systems that drifted there quietly, the ones that arrived embedded in vendor platforms, that migrated from research prototypes to operational decision-making, that staff adopted informally because the tool was useful and nobody said they couldn't: those systems are often higher risk and completely ungoverned. The governance attention ends up inversely proportional to the actual risk. It's Rasmussen's migration.
So you have multiple systems, limited time, no governance structure, and you need to figure out where to focus first. And that's a triage question, that in itself needs a triage method.

Some existing approaches that don't quite work for me

Before I describe what I use, it's worth understanding how a few frameworks handle prioritisation, because you'll encounter them in practice and they each have their uses.
Microsoft's Responsible AI Impact Assessment can be useful. It starts with stakeholder identification and then works through harms across six Responsible AI principles. The template is publicly available and the stakeholder-centred thinking is its real strength. But completing one properly for a single system is quite a large piece of work. It mentions prioritising based on frequency and severity, but doesn't really give you a concrete method for rapid prioritisation when you're looking at multiple systems simultaneously.
ISO/IEC 42005, published in May 2025, is the first internationally standardised impact assessment methodology. So it has structured processes, a harms taxonomy, and integration with ISO 42001. It's comprehensive, thorough, and designed for formal documented assessments in mature compliance-heavy governance programs. If you tried to conduct a full 42005-compliant assessment for each system in an organisation that doesn't even have an AI inventory yet, you'd be months into the work before you had a single actionable finding. Sorry, but I'll pass on 42005, especially for triage.
Then there's the EU AI Act, which takes a different approach entirely. It pre-classifies systems into prohibited, high-risk, and lower-risk tiers based on use case. I've written at length elsewhere about my frustrations with this approach, but the relevant point here is that the Act is frequently described as "risk-based" when it is really product categorisation-based. It assigns a tier based on the use case category, before anyone has examined the actual system, its context, or the state of its controls. 
The problem is that AI systems in organisations don't always behave like products. They drift, exactly as Rasmussen described, from research to operations one reasonable step at a time. They get embedded in vendor platforms. They get adopted informally by staff. The Act assumes intentional design, intentional deployment, and an identifiable provider. But the research tool that quietly became an operational decision-maker was never "placed on the market" by anyone. It just drifted there. So the EU AI Act, and it's absurdly complex classification model is of no use whatsoever for triage of real risk and prioritisation of effort. 
What these approaches share is that they were designed for thoroughness rather than speed. They might be valuable for formal assessments and mature programs, but none of them answer the question you face on day one: multiple systems, no structure, limited time, where do I start?
Not a new problem of course. Sarah Clarke, a self described 'AI Realist', and a wonderfully intuitive AI governance practitioner in England, wrote about encountering the same kind of challenge in supplier governance. She describes being asked to assess 600 suppliers. After checking what everyone actually meant by "supplier," it turned out to be 2,000 entities. No army of consultants, no unlimited budget though. And the prioritisation guidance she received was: focus on what's "critical" or "material." Her first question: what exactly do you mean by critical? And the answer: "It depends".
She vividly describes a particular circular dependency that anyone who's done this work will recognise. You should prioritise based on risk, sure. But risk assessment requires context. Context requires specialist effort. Specialist effort requires prioritisation. You're going in circles before you've started.
One of her observations is interesting: if you don't bother doing due diligence for anything below a certain threshold, you will miss important things. Like the ten-person startup that gets access to all your CRM data on a shoestring contract. Or in AI governance terms, the AI system that slipped in below everyone's radar because it wasn't expensive enough, visible enough, or "product-like" enough to trigger the formal process. I recommend her articles and approach to use case triage - they're worth a read.
So what I'll describe is my approach - it shares some bones from these more complex frameworks, leans on Rasmussen's ideas of drift to danger, and shares some similar perspectives to Sarah's. Ultimately, it's just about triage - a really quick way to identify where to focus on Day 1.
Let me walk you through it. Hope it helps.

Rapid harm triage

The method I use has four steps. It's simple enough to apply in a workshop or on a whiteboard, practical enough to use on your first day in a real governance role, and just enough structure to be defensible rather than gut feelings. I'm not trying to replace the comprehensive frameworks, maybe later they're applied. It's rapid, defensible prioritisation in a day.

Step 1: Identify who is affected

For each AI system, quickly identify four stakeholder groups:
  • Direct users: the people who interact with the system. 
  • Subjects: the people the system makes decisions about or generates predictions for, often different from the users and often the most vulnerable. 
  • Affected others: the people who don't interact with the system but are affected by its outcomes. 
  • And the organisation itself: reputational, regulatory, legal, and financial exposure.
The distinction between users, subjects, and affected others matters. An AI system that recommends courses to university students has the student as both user and subject. But an AI system that predicts which patients should be prioritised for specialist appointments has clinicians as users and patients as subjects. And the patients who don't get prioritised, who never know an algorithm played a role in how long they waited, those are affected others. Frameworks that focus only on the user-system interaction systematically can miss harm to the people who never interact with the system at all.

Step 2: Generate harms through six types of impact

Once you've identified the stakeholders, you need a structured way to surface the specific harms each group might experience. For each stakeholder group, I run through six types of impact and ask whether this system could affect this person's safety, privacy, autonomy, fairness, access, or trust. You could use Microsoft's 6 RAI dimensions too, or come up with your own. These 6 just work for me:
  • Safety: could this system affect someone's physical health, clinical safety, or bodily wellbeing? 
  • Privacy: could it expose, misuse, or lose control of someone's personal information?
  • Autonomy: could it undermine someone's ability to make informed decisions, to give or withhold consent, or to understand what is happening to them? 
  • Fairness: could it treat some people less favourably than others, or produce systematically different outcomes for different groups? 
  • Access: could it affect someone's ability to receive services, support, or opportunities? 
  • Trust: could it damage the relationship between the person and the institution?
These map broadly to harms taxonomies you'll find in ISO 42005 and the NIST AI RMF too, but they're just framed as questions about people rather than categories of technical failure. You're asking "could this person be harmed?" not "could this system fail?"
Now, I don't need to find a harm for every impact type and every stakeholder group. Some combinations won't apply. When a question surfaces a specific harm, write it down and move on. You need enough structure to avoid missing the important ones, not enough to produce exhaustive documentation. And you may well come up with harms that don't neatly fit into those six. That's fine, this is for triage purposes.

Step 3: Characterise each harm on three dimensions

This is where it gets interesting, and where I think the approach diverges most from what's common in AI governance practice. For each harm you've identified, you assess three things: severity, breadth and vulnerability, and detectability.
Severity is pretty straightforward. If this harm occurs, how bad is it for the person affected? I use a four-point scale: inconvenience, significant distress, serious harm, or life-threatening. You're judging consequence magnitude, not probability. Don't get into the effectiveness of risk mitigations, etc - you're thinking about potential severity if the harm occurs.
Breadth and vulnerability asks how many people could be affected and whether they're in a position of vulnerability. A harm affecting one user differs from one affecting every customer. And vulnerability amplifies severity. Children can't advocate for themselves. Patients in acute care can't opt out. People with low digital literacy can't interrogate the system's outputs. So when AI systems affect vulnerable populations, the severity of any harm is amplified by the limited capacity of those populations to detect, report, or avoid the harm.
And then detectability. This is the dimension that, in my experience, genuinely matters most for governance prioritisation and is most consistently overlooked. 
It's not very common in AI governance (at least in my experience) to see practitioners factor detectability into their risk assessments. You might not have even heard of it, unless you come from a safety engineering or quality engineering background. Detectability comes from a method called Failure Modes and Effects Analysis (FMEA). It's a method that has been used in manufacturing, aerospace, automotive, and medical device industries for decades to prioritise risk based on three factors: the severity of a failure's effect, the probability of it occurring, and the probability that it won't be detected before reaching the customer. That third factor, detection, is what makes FMEA quite distinct.
You see the standard risk matrices you're probably familiar with give you severity times likelihood, and that's it. FMEA adds: would you even know this was happening?
And that addition can change the priority completely. A severe harm that is immediately obvious is still a problem, but it triggers a response. The system goes down, people mobilise, the incident gets handled. A severe harm that is invisible is a problem that grows. It accumulates quietly over weeks or months. By the time someone notices, the damage has compounded and may be very difficult to reverse. 
Let me give you a few examples. Say an AI chatbot gives bad medical advice to a patient's family. The family follows that advice, the patient's condition worsens, and they present to emergency. The harm is potentially severe, but at least the feedback loop is there. The problem becomes visible and someone can act on it. Now think about an AI system that predicts risk scores for adolescents with mental health conditions, and those scores quietly shape which teenagers get additional counselling, priority psychiatrist access, and crisis support. The scores are wrong for teenagers from certain cultural backgrounds. Nobody has validated the model for this population. There's no monitoring. The harm of teenagers not receiving support they need, accumulates invisibly. Nobody involved even knows an algorithm is shaping the outcome.
Same severity. Radically different detectability. And you want your governance effort to reflect that difference.
I think that missing detectability as a factor, is one of the reasons that AI governance work gets prioritised on the wrong problems. The chatbot is visible, so it gets scrutinised. The clinical risk prediction system is invisible, so it gets nothing. Governance attention flows to what can be seen. Detectability tells you where to look for what can't.
Rasmussen's migration model, is then fundamentally about realising that the most dangerous drift is the drift you can't see. Dekker's concept of practical drift captures it from another angle: the slow, normalised divergence between how work is designed and how work is actually done. But the common thread across them both is that the most dangerous failures are the ones that don't announce themselves. Safety science has understood this for decades. AI governance is slowly starting to catch up.

Step 4: Prioritise

Ok, now bring the three dimensions together. The pattern tells you where to focus.
  • Immediate: high severity, vulnerable population, low detectability. These are the systems where serious harm can accumulate invisibly. They need governance attention now, not because the probability of harm is necessarily highest, but because you have no mechanism to know whether harm is occurring right now. You're flying blind, and the people affected can't tell you. The AI system that quietly shapes which teenagers get access to mental health support, with no monitoring and no validation for the population it's scoring, sits here.
  • High: high severity, but high detectability. The feedback loop exists. Harm would likely be noticed, reported, and trigger a response. These systems still need governance, but the urgency is lower because the organisation has some natural capacity to catch problems. The chatbot that gives bad medical advice is serious, but the patient's deterioration is visible to clinicians. There's a path from harm to detection to response, even if that path needs formalising.
  • Moderate: lower severity with broad reach and low detectability. Each individual impact is small, but the system affects thousands of people and the harm accumulates invisibly. An AI scheduling system that subtly disadvantages certain demographic groups in appointment allocation might fit here. No single patient is seriously harmed, but the pattern across thousands of patients is inequitable and nobody is measuring it. These systems are easy to overlook because no individual complaint seems alarming. The aggregate is what matters.
  • Low: lower severity, broad reach, high detectability. The harm is individually minor and the feedback loops work. People notice, they complain, the organisation can respond through existing channels. These systems need governance eventually, but there's likely no urgency. Schedule them for the next review cycle and move on.
  • Defer: low severity, limited reach, high detectability. The internal chatbot that occasionally gives slightly unhelpful answers about office policies, and everyone knows it and works around it. This is like a green tag. Formally noting that you've assessed the system and decided it doesn't warrant governance attention right now is itself a governance act. It's defensible, and it prevents the "everything needs governance" paralysis that stops practitioners from making meaningful progress on the systems that actually matter.
The whole point of triage is that some things don't get treated yet, and that's a defensible decision. Without that lower tier, you're back in the trap of treating every AI system as equally urgent, which means nothing gets the attention it actually needs.
There's a sixth category, but it sits alongside the priority stack rather than within it. This is the shadow AI finding: unknown severity, unknown breadth, very low detectability. You can't assess the risk because you don't know what the systems are, what data they're processing, or what decisions they're influencing.
This isn't a system priority. It's an organisational gap. The five categories above answer "which system do I focus on first?" The structural finding answers a different question: "can I even do this triage properly?" If you're finding that significant shadow AI exists, the answer is no, and you have to do something else. Build the inventory mechanism, the policy, the discovery process, that makes future triage possible. You can't triage what you can't see.
This connects directly to the diagnostic method I wrote about last week. Once you've triaged and identified which systems need attention first, you stand in the chalk circle and diagnose the mechanisms. The triage tells you where to stand. The diagnostic tells you what to fix. They're complementary practices of course.

Defensibility and what this triage doesn't do

One of the reasons I prefer this structured approach over gut instinct is defensibility. When you're the governance practitioner making a recommendation to leadership about where to invest limited resources, you need to be able to explain why you're recommending System X over System Y. "I used a structured triage that assessed severity, vulnerability, and detectability for each system, and System X presents high-severity harm to vulnerable subjects with no detection mechanism" is a defensible recommendation. "I had a bad feeling about System X" is not. "System X is agentic, while other systems are just plain old machine learning" is definitely not.
Now, it's also important to be honest about the limitations. This is a rapid triage, not a comprehensive impact assessment. It produces defensible priorities, not definitive risk quantification. It helps you decide where to focus first, that's all.
You may still need the comprehensive assessments later. Some practical version that might bring elements of ISO 42005 or Microsoft's Impact Assessment template is genuinely useful for deep-dive analysis of a single system. The EU AI Act's classification system has legal force and requires compliance regardless of what your triage says - even if the practice of 'high-risk' classification is a pointless exercise with no practical safety value (in my opinion). This triage approach doesn't replace any of those. It answers a specific question they don't: given everything you're looking at, where should you start?
And of course, it requires judgment. The four-point severity scale, the assessment of vulnerability, the evaluation of detectability, these are all judgment calls. Different practitioners will make different calls. That's completely fine. The structure helps to make sure you're making those judgments explicitly and consistently, not implicitly and unevenly. As you learn more about each system, as you stand in the chalk circle and observe, and as you talk to the people who use and are affected by these systems, your assessments will change. 

Wrapping up

So to wrap up this article - here's my suggestion: the first thing you do when you walk into a new organisation is not to design a framework, or map the EU AI Act, or write a policy. The first thing to do is figure out what AI systems exist, who they affect, and which ones are causing harm that nobody can see.
That triage, done well, prevents the weird governance inversion of spending too much effort on systems with the lowest real risk. If the method from last week's article was about standing in the chalk circle and watching, you might take this one as being about figuring out which chalk circle to stand in first.
This kind of triage thinking is central to what we teach in the AI Governance Practitioner Program, particularly in the Foundation Track Cohort where participants apply it to realistic case scenarios. I share this method not because I think everyone should follow my specific approach. Governance is a craft. Practitioners develop their own methods. But the practice of rapid, structured prioritisation is a skill that can stop you from drowning in the complexity of "everything needs governance" and let you make a meaningful difference with the time and resources you actually have.
If you've developed your own approaches to this triage problem, I'd genuinely like to hear about them. I really suggest you check out the approach that Sarah Clarke describes here. The more methods we share, the better we all get at this work.
You can always find out more about our AI Governance Practitioner Program here, and always welcome to join if you'd like to learn these practices in depth.

References

  • Rasmussen, J. "Risk management in a dynamic society: a modelling problem" (Safety Science, 27(2), 183-213). https://www.sciencedirect.com/science/article/abs/pii/S0925753597000520?via%3Dihub
  • Microsoft Responsible AI Impact Assessment Template and Guide (2022). Template: https://blogs.microsoft.com/wp-content/uploads/prod/sites/5/2022/06/Microsoft-RAI-Impact-Assessment-Template.pdf
  • ISO/IEC 42005:2025. "Information technology - Artificial intelligence (AI) - AI system impact assessment." https://www.iso.org/standard/42005
  • Dekker, S. (2011). Drift into Failure: From Hunting Broken Components to Understanding Complex Systems. Ashgate. I
  • https://www.linkedin.com/pulse/ai-use-case-triage-origin-story-sarah-clarke-hgzwe/?trackingId=dqGc5qYaSi63tT34iJ8jXA%3D%3D