Clinical AI Beyond Runtime Inference

I. The model race has mistaken progress for readiness.

The pressure to "ship" clinical AI products is palpable, and the pace is dizzying. The latest models improve through an empirical cycle of training, post-training, evaluation, safety testing, controlled release, monitoring, incident investigation, and revision.^1–2 No pre-deployment process can itemize every failure that can emerge in use.

In clinical care, the threshold for a product to be "ready for launch" is much higher. To be trustworthy in the day-to-day of healthcare, a model must deliver measurable value under real-world conditions. Before deployment, these questions need clear answers:

• What is the intended use of the product, and for whom?

• At what stage does it enter the clinical workflow?

• What does it aim to improve compared with how care is currently delivered?

• What decisions can it impact?

• What is its behavior when the context is missing, in conflict, or outside the scope intended?

• Who is accountable for the model’s errors (those caught, and those no one catches)?

• What harms could follow?

• Who is expected to perpetually identify and override its outputs?

• Can the model’s mistakes be inspected retroactively?

• How is responsibility divided between clinicians, institutions, and developers?

Warnings that "AI can make mistakes" and terms of use shifting responsibility onto clinicians to catch those mistakes leave these questions unanswered and undercut the merits those products set out to prove^3–4. There is an appreciable risk of clinicians becoming desensitized to these warnings. When something performs well most of the time, reliance on it becomes a treacherous habit.^5–6

Clinicians should not bear the burden of detecting errors within an otherwise convincing process; value can be created without relying on perpetual oversight as the primary safeguard.⁷

Challenging cases with no obvious or knowable right answer can and should draw on the most capable models at inference time. However, once a recommendation is approved for a given context, its application should be immutable, testable, versioned, and traceable, rather than being generated each time afresh.

Responsible development is incomplete until the model is also tested against the clinical standards it is expected to follow, in a form that can be executed deterministically. Testing behavior across a wide range of outputs improves the system over time but does not answer the upstream design question: what should clinical AI get right by default?

II. The limits of progress measured by benchmarks

Model rankings in use do not measure clinical performance. Thousands of studies now compare models against a wide variety of clinical tasks and benchmarks, including general medical question answering, open-ended health dialogue, individual physician-authored tasks, multilingual clinical text, long-horizon electronic health record (EHR) based tasks, difficult diagnosis, trials evaluated against typical clinical practice, and detection of potential harm.^8–17 These studies show how fast capability is advancing but also how differently people measure it.

Benchmarks turn into a media cycle that declares a new winner every few days. The playbook goes: publication, a headline result, close examination of the cases and methods, substantive rebuttal, and then the attention moves to another study. Too little attention goes to what must be true for a product to be trusted in clinical care.

Having contributed as a physician advisor to HealthBench and HealthBench Professional^9–10, I saw their value in turning broad impressions of health performance into specific, contestable measurements. However, they leave an unanswered question: which aspects of clinical care should not depend on another sampled output.

The June 2026 Nature Medicine comparison of three frontier models with OpenEvidence and UpToDate Expert AI found the general-purpose models ahead in their cases, interfaces, graders, and outcomes.¹⁸ A subsequent preprint, using OpenEvidence queries, different model versions, and a different physician-grading design, found OpenEvidence ahead.¹⁹ The ongoing disputes expose how much a score relies on choices about cases, tasks, model versions, and scoring methodology, of which little is immune to commercial incentives^20–21. A leaderboard is closer to clinical equipoise than to a verdict, and medicine knows what to do with equipoise: run the trial.

III. The neglected object of progress

The dominant object of progress in clinical AI today is runtime performance. The neglected object of progress is executable clinical knowledge that can accumulate, stay current, and be invoked non-probabilistically.

Computable clinical knowledge has long been proposed as a reusable object in its own right.²² WHO SMART Guidelines provided a partial foundation for this work by addressing structured representation, localization, implementation, updating, and reuse.²³ But important work remains: reconciling conflicting authorities, preserving local clinical variation, testing interactions, seeking approval or modification from governance, and activating each usage of its version.

RAG improves access to relevant evidence at runtime but does not change what produces the answer: the retrieved object is handed to the same probabilistic model, which still has to read it against the case. Even with the exact guideline documents in front of it, answers dropped details, repeated outdated cutoffs, merged older and current recommendations, and changed from one run to the next. ^24–26

A published finding does not, on its own, become a clinical action. Evidence-based medicine (EBM) combines evidence, clinical expertise, and patient values as distinct inputs to care decisions.²⁷ GRADE makes another distinction explicit: certainty in an estimated effect is not the same as the strength of a recommendation.²⁸ Between evidence and care lie interpretation, adoption, local feasibility, and judgment.

Clinical knowledge can be deconstructed at a much more granular level than an entire guideline. When the decision has been examined and accepted for a given scope, the controlling logic becomes a reviewed clinical resolution, and its versioned, executable form is the artifact.

An artifact represents one atomized item of executable clinical knowledge. What surrounds it is the chain that must hold: the correct version and local branch invoked, patient data mapped correctly, the result presented within the intended workflow, and exceptions and conflicting recommendations handled.²⁹ Authorization, monitoring, and revision belong to the same chain. A fixed rule can still lead to the wrong action if an inappropriate local version is selected or its result appears in the wrong setting.

The work involved in creating and keeping such artifacts is considerable: the recommendations have to be interpreted, the exceptions kept, the local data mapped, the conflicts settled, the tests prepared, and the approvals documented.³⁰ The alternative is for clinical AI applications to remain dependent on runtime inference for logic that could be made inspectable and deterministic.

IV. Creativity is not the enemy

Right after the launch of ChatGPT in late 2022, I tasked GPT-3.5 with writing a haiku about a dog fixated on sparkling water (for no particular reason). I will never forget the rush of discovery and sense of something shifting in the world (much more than the actual lines it produced).

Discovery and excitement should survive in clinical AI. Generative freedom should stay broad where variation matters.

1. In a puzzling diagnostic case (obscure presentations, an odd pattern of symptoms), models can produce helpful hypotheses, synthesize scattered findings from voluminous patient records, challenge the premise of a diagnostic label, and help pinpoint which question, exam, or test is most likely to be helpful. Even here, inference is bounded by time limits within the specified pathway, validity criteria, contraindications, and criteria for avoiding important omissions.

2. Whereas there are several ways for clinicians to draft notes, there are fewer correct ways to report medication dosages, lab test results, and the presence or absence of red flag symptoms. Where the record is uncertain or contradictory, the output should preserve that uncertainty.

3. Flexibility is essential in communicating with patients. An appropriate decision may require distinct explanations depending on the individual's health literacy, experiences, phobias, and values. A model can help develop the right kind of questions and support translation, comparison, instruction development, and portal messages.

4. Large language models can handle an abundance of information that would otherwise be nearly impossible to process manually. They enable exploration of records, creation of timelines, exploratory anomaly detection, retrieval of relevant literature, and identification of trials requiring further investigation. Trial matching is a good example of cooperation between generation and retrieval.³¹ Generation and retrieval together expand the pool of candidates and correlate raw records with complex clinical trial descriptions. But generation and retrieval cannot, without violating criteria, implicitly extend inclusion/exclusion criteria or fill information gaps.

5. Models can work across scientific literature, molecular data, experimental results, and clinical observations to propose mechanisms, identify repurposing candidates or potential targets, and suggest experiments to distinguish competing hypotheses.³²

Executing standards of care that have been made explicit and approved need not remain probabilistic.

V. Correctness is contextual, and it should accumulate

The context for a care decision commonly includes: (1) the evidence upon which a decision was based; (2) the various authorities (e.g. guideline developers, payers, formulary committees, etc.) that were relied upon; (3) local rules and context, including coverage and reimbursement; (4) the patient's clinical facts, including current condition, history, treatments, and observed data; and (5) the patient's preferences and goals.

That "faithfully" executed recommendation then has to match the real person in front of us. The health record is a still frame from the moving picture of the person's life (and sometimes an old one). A conversation with the patient can update that record. It can provide information about a prior reaction to a medication or situation, a medication that was stopped months or years ago but not updated in the EHR, a trade-off the patient declined, or a referral that was not activated. These newly elicited facts specify which branch of the explicit clinical logic applies.

Where and when care happens should not be an afterthought in implementation. A rule for treating a condition, contained in a nationally valid guideline, might require that a less expensive treatment fail before this treatment can be covered by public funds. The local branch must encode that difference.

Guidance from large bodies such as the World Health Organization (WHO) still has to go through adoption processes at the local level, since local data, available resources, the actual situation within organizations, feasibility, and equity all affect how recommendations are implemented.³³

However, reusing knowledge for other cases of the same or different diseases needs an update path. Guidance changes, and recommendations written for one condition can conflict when combined with another.³⁴

The ceiling for deterministic clinical AI applications will extend far beyond calculators. It will execute clinical logic spanning complex pathways. The underlying logic can (and should) be as complex, branching, locale-agnostic, and patient-specific as one can imagine, including cases where several guidelines and local policies overlap. Once reviewed and adopted for a defined scope, its complexity is no reason to perpetually reconstruct it through fresh inference.

VI. A foundation without false certainty

For a specified scope, an approved clinical resolution should be deterministic and apply the same logic across millions of repeated queries. In guidelines, terms such as "consider", "usually", and "in selected patients" need to be retained to preserve the discretion authors declared.³⁵ Reviewed clinical resolutions become candidates for an unambiguous representation when all five requirements are met:

1. The problem recurs frequently enough to warrant retaining the resolution.

2. The resolution has an explicit governing authority

3. The inclusion/exclusion criteria, expected outputs, and exceptions are specified

4. It is possible to verify both its origin and its behavior within its stated context.

5. A human can authorize, localize, version, monitor, modify, and remove the artifact.

Uncertainty and contention need not be eliminated. Various credible authorities may be explicitly identified by name and published source (e.g., scientific article, guidelines document). A question may be posed when data or values are unknown. Missing inputs, an out-of-scope case, the absence of an adopted resolution, or an unresolved conflict between applicable resolutions should each produce an explicit non-answer rather than one arbitrary path put forward convincingly.

Formalization does not mean an artifact or its test cases are unerring. It means that within a declared scope, the approved behavior is reproducible before and during use.³⁶ Verification concerns assessing whether the artifact conforms to that specification. Validation asks whether the specified behavior is fit for its intended clinical use.³⁷ The claim is therefore bounded: what the artifact is designed to do, under which conditions, and which uncertainties remain outside its scope.

The data path belongs inside that claim. A correct dosing formula can still produce the wrong result if, for example, the weight carries the wrong unit, a discontinued medication remains active, or the deployed artifact is no longer the approved version.

Clinically established and consistently applicable artifacts should be presented to clinicians in a ready-to-use form. Conflicting authorities should be shown with their provenance. Missing information should trigger a question. While clinicians should be able to inspect every step, perpetual clinician verification should not be the primary safeguard.

A named owner must review changes to the source material, decide which local branches remain up to date, and publish revised artifacts. The owner must keep a record of changes to the source material and resulting artifacts. When a defect or source change invalidates an artifact, the owner must withdraw, disable, roll back, or place the affected release in a controlled degraded state as appropriate. Testing must replicate all aspects of data mapping, interface behavior, clinical workflow, required staff, and local resources or supervision.²⁹ End-to-end testing of the tool as it will be used in practice must demonstrate that deploying this artifact performs as intended, controls foreseeable hazards, and does not simply shift risk elsewhere in the workflow.

We ought to use improvements in general-purpose models to improve care delivery rapidly, subject to medicine’s existing checks and balances. Executable clinical knowledge should be safe and reliably invocable, deterministically, across settings, without erasing what varies.

Clinicians should be able to see what was carried forward, what was inferred, and where the two conflict. This must apply to patients as well, who, at a different scale, are much more vulnerable to what might appear to be a convincing process. The pace of medical discovery will continue to accelerate. To the benefit of humanity, it is now time to make more of the passage from discovery to practice non-probabilistic.