Safe, fast and trusted, but who is checking the “safe”?

What the Commission gets right

The Commission diagnoses one of the central problems with the current regulatory model: it was largely designed around products that can be assessed at a particular point in time.

AI is different.

Performance can vary between environments, products get updated, and models change. The way a technology is used can also change its risk profile.

The Commission therefore calls for a more proportionate, lifecycle-based approach, including monitoring after deployment and the use of real-world evidence. I believe that is the right direction.

Its second chapter also gets a lot right. The Commission recognises that safety cannot sit entirely with the manufacturer or entirely with the clinician using the product. It describes the risk of a “liability sink”, where accountability can end up with healthcare professionals who may not be able to see or control the risks created elsewhere in the system.

The answer is shared responsibility, which recommendations 26 and 28 address by seeking clearer allocation of responsibilities and risk controls between manufacturers and providers, including through contracts. To me, that starts to look like the DCB0129-to-DCB0160 handover, extended across the AI lifecycle.

The report also gets some smaller but important things right. It proposes recording device versions in patient records to support outcome auditing, greater transparency about dependence on general-purpose AI models through procurement, stronger enforcement mechanisms, and a proportionate approach to informing patients when AI is used in their care.

And its focus on organisational readiness is welcome. The Commission proposes an AI readiness toolkit and recognises that healthcare organisations need the capability to deploy, monitor and govern AI safely, rather than simply being given a product and told to use it.

So where does it fall short?

The standards problem

The report proposes an “AI readiness toolkit” to help create a shared language between manufacturers and healthcare providers. That is useful, but we should be careful about starting from scratch.

The report does not meaningfully engage with established AI governance and assurance standards such as ISO/IEC 42001, BS 30440 and ISO/IEC 38507. It does reference DCB0129, DCB0160 and DTAC as existing NHS assurance routes, but the wider standards landscape is largely absent, which is a noticeable shift compared to non-AI forms of technology.

I am not suggesting that certification against a standard makes an AI system automatically safe, but standards can provide a common structure for governance, assurance and audit. They give organisations something against which processes can be tested, rather than relying entirely on self-assessment.

The Commission itself says its proposed toolkit is intended to create a shared language between manufacturers and providers. We already have standards that can contribute to that shared language. The opportunity is to bring them together, not create another parallel framework.

AVT exposes the boundary problem

The Commission uses AVT as useful test case of how AI products can contain multiple functions, some of which may fall within medical device regulation and some of which may not. It recommends a more function-based approach, rather than treating an entire product as regulated simply because one function has a medical purpose.

The MHRA's July 2026 guidance makes the position clearer. Some AVT products, including those intended solely for transcription, summarisation, drafting correspondence or suggesting clinical codes for review, are not medical devices. Others, such as products that provide diagnostic or treatment suggestions or take clinical actions without appropriate review, can be medical devices.

I’ve already shared my views on the MHRA’s particular decision and am left wondering what the minimum assurance floor is when an AI product is not a medical device, but its output can still influence patient care.

In my view, a summary that a clinician uses as part of ongoing care sits within the wider decision-making chain, even if the software is intended to perform an administrative function. NHS England's guidance already points towards an answer for ambient scribing. It expects organisations to complete DCB0160 documentation, including the safety case, hazard log and monitoring framework, and suppliers to have completed DCB0129. It also warns that generative AI products can introduce unintended functions and that users may be able to prompt products beyond their intended purpose.

That is why the boundary between “regulated” and “not regulated as a medical device” cannot become the boundary between “assured” and “not assured”.

The legal test is still the product's intended purpose, and I am not arguing that every summary tool should automatically become a medical device, but that if a product can affect care, there needs to be a clear assurance route for it, regardless of which regulatory category it occupies. That is even more important as products evolve. The assurance floor should not depend entirely on which side of a regulatory boundary the product happens to sit on at a particular point in time.

Verbs without a clock

Recommendations are only useful if someone is expected to act on them.The report contains a lot of language such as “should consider”, “should explore” and “could consider”. Recommendation 33, on training by healthcare providers, is notably stronger: it says providers must ensure that professionals and wider staff are appropriately trained.

But across much of the report, there are no implementation dates and no clear timetable for turning the recommendations into practice.If we want organisations to build confidence in AI, they need to know what is expected of them, when and by whom – so I hope the cross-government response will give us more certainty on timelines.

Shared responsibility needs capability

The Commission is right to describe AI safety as a shared responsibility, but shared responsibility only works if the people carrying it responsibility have the capability to do it.

The report has a strong focus on AI literacy and training for healthcare professionals.

Recommendation 33 requires healthcare providers to ensure staff are appropriately trained to use the AI technologies deployed in their practice.But that is not the same thing as building the capability to assure AI.

The people responsible for assurance in practice also include Clinical Safety Officers, information governance teams, procurement professionals, digital leaders and boards. I would have liked to see a much clearer articulation of the roles and capability needed to make that governance work.

This is particularly important because DCB0129 and DCB0160 already place clinical safety responsibilities on manufacturers and deploying organisations. NHS England's current review of those standards has itself identified gaps around AI governance, complex system interactions, cross-organisational working and post-implementation monitoring. We should use that opportunity.

We should not repeat the safety assurance winter

There is a risk here that we repeat some of the history of clinical safety standards.

DCB0129 and DCB0160 were introduced to bring systematic clinical risk management into the development and deployment of health IT. They have since become an established part of NHS digital safety, but awareness, capability and implementation have taken time to develop.

But AI is moving much faster. Some organisations are already developing mature approaches to AI governance. Others are still working out what assurance should look like.

If the UK wants to accelerate AI adoption in healthcare, assurance cannot be something we retrofit once problems emerge.The Commission is right that regulation alone cannot deliver safe AI. Its own report describes the need for a collaborative, system-wide approach involving manufacturers, providers, professionals, regulators and policymakers. The next step is to make that responsibility demonstrable.