We Don’t Treat Every Patient the Same. Why Do We Treat Every AI Tool That Way?

To govern healthcare AI safely and effectively, we need a risk-based approach—one that mirrors how clinicians already think.

“Let’s add it to the time out”

This sentence predictably came at the end of every meeting I’ve ever been to after something went wrong in the operating room. Every time, I inwardly groaned (after five years as the Medical Director of the OR, I’m sure some of those were more audible than I planned). Time outs are great - they have good evidence to support improvements in care, and are now completely ingrained in every operating room. Starting a case without one would be like forgetting to put a mask on before entering the OR. It’s a habit.

Note: This is not a photo of me in green; I would be wearing my battery-powered heated vest in any photo ever taken in the operating room.

But in the past several years, the time out has become a convenient receptacle for any problem that has ever happened, or people think might happen. So the knee-jerk reaction is to add more components to the time out.

I’m sure you can guess what happens next.

After a while, the “extra” elements are ignored completely. There are too many of them, many are not relevant for the specific case that’s about to take place, and people don’t really understand the point of them. The too-long checklist becomes background noise.

Just like overloaded time outs lose their value, sprawling governance forms can obscure the truly important questions. Their goal is similar to that of surgical time outs: to capture and convey crucial information to reduce risk. This is a good idea! We want AI governance committees to do some due diligence about the new healthcare AI products being introduced. But we need to make sure the process doesn’t get so mired in details that the goal of the requirements is lost.

What Is AI Governance, Really?

“AI governance” gets tossed around a lot these days, in everything from specific administrative tools to catastrophic risks that people worry could lead to the end of the world.

In healthcare, it refers to “the practice of reviewing, assessing, and evaluating individual AI tools to ensure that they can be used safely, responsibly, fairly, and effectively with the health system’s patient population and in compliance with applicable laws”.

Depending on your role in the healthcare system it might mean legal oversight (for regulators), IT infrastructure and risk management (for hospitals) and transparency (for patients).

For frontline clinicians, it’s a lot simpler. We want to know:

• Can I trust this tool?

• What happens if it’s wrong?

• Who’s responsible?

Let’s be clear: this isn’t just a paperwork problem. Deploying an AI tool without proper governance can lead to automation bias, where clinicians over-trust model outputs. It can widen health disparities if a model underperforms in certain populations. And it can create real harm if no one is monitoring how the tool behaves once it's deployed.

AI governance is multidisciplinary and culture-dependent

AI governance spans the broad fields of law, ethics, IT, clinical care, engineering, and sociology, with overlapping domains as in this paper on AI ethics:

How a hospital approaches AI governance depends mostly on culture. AI decisions are often made locally, shaped by who’s in the room, who’s asking questions, and how many headlines a hospital wants to avoid. If your organization has a low threshold for any errors, you are going to be very slow to approve and adopt new software. You will want to make sure that if you ever have to explain why you decided to approve something, you will be able to defend your decision. If your organization wants to be a first-mover with AI technology, you know you’ll need to explain why you’re the bottleneck in the process.

The lack of a standardized process means that the decision-maker, who is often a physician, has to take much greater personal risk. They can’t rely on “following the system” or the “AI Governance Standard of Care” to protect them from someone second-guessing their decisions.

Without clear standards, both developers and decision-makers are left improvising—and absorbing more risk than they should.

“It’s either one question or a million”

The current state of AI governance in healthcare is a patchwork at best. Most hospitals now have some kind of AI governance committee. Sometimes these are housed within an existing IT review committee, procurement program, or clinical review committee. Often there is representation from IT, legal, data science, compliance, and procurement departments. The CIO, CTO and/or CMIO are almost always on the committee.

A year ago, only 11% of hospitals said they had AI-specific governance processes. I’m sure that number is higher now, but clearly they are still new and evolving.

Healthcare AI developers often tell me how variable the process is for both governance and evaluation. Some hospitals are satisfied if the product has gone through literally any testing process one time, while others require multiple rounds of evidence and committee meetings, often with additional information being requested on an ad-hoc basis.

The variability in these processes makes it hard for vendors to know what to expect and adds uncertainty, cost, and time to integration.

More importantly, the variability in AI governance requirements is a symptom of the fact that standards are still evolving, and no one really knows what they should be doing.

Could the IRB be a model for AI governance requirements?

Just like the OR time out creates a structure for everyone to align before doing something high-risk, AI governance should create a way for a multidisciplinary team to ask critical questions. But unlike the time out, which is short and targeted, AI governance forms are increasingly sprawling and generic.

Right now, most AI governance forms ask dozens of questions but don’t provide a structured way to determine clinical risk. A non-clinical chatbot is often reviewed using the same form as a predictive model used to prioritize ICU beds. Just as in time outs, the most critical questions are easily overlooked or ignored in the flurry of non-pertinent data.

(Note that I’m explicitly discussing clinical risk here, not legal or security risks, though those aspects could use similar approaches)

AI governance requirements should function more like the tiered, risk-adjusted review system of an Institutional Review Board (IRB):

• Low-risk tools (e.g., administrative or educational models) might go through expedited review.

• Moderate-risk tools might require documentation, clinical validation, and monitoring plans.

• High-risk tools, especially those embedded in clinical decision-making, should trigger a full review, including subgroup performance analysis, post-deployment monitoring, and clear fallback procedures.

This kind of governance would allow for appropriate AI oversight without drowning in a sea of questions, and allowing these committees to spend their valuable time focusing on higher-risk implementations.

We don’t need longer forms—we need smarter questions. AI governance should help us pause at the right moment, not bury us in paper. If AI is going to become part of clinical care, the systems we build to oversee it must be as thoughtful and intentional as the care we provide.

Does your hospital have a formal process for evaluating AI tools? If you’re a clinician, do you know who approved the last model you used, and did you receive a copy of relevant information from the AI governance form?