← Case studies
CASE STUDY · HEALTHCARE

Healthcare AI toolkit: what the audit found, and how we fixed it

We ran our audit tool on a healthcare AI toolkit. It is a library that teams build clinical-adjacent applications on. This case study sets out what the audit found, what it means for a team that builds on the toolkit, and what our forward-deployed engineers did after the report.

CASE STUDY 05 HEALTHCARE EU AI ACT HIGH-RISK ISO/IEC 42001
  • AI components detected 5
  • Findings 5 All medium
  • Tier High-risk
  • Obligations checked 28

At a glance

Sector
Healthcare software and clinical-adjacent tooling
System
A Python toolkit for healthcare AI. It includes examples for discharge summaries, question answering over clinical records and clinical coding.
EU AI Act scope*
The tool places the toolkit in the high-risk tier. The tier depends on the intended use of each application built on the toolkit.
AI components detected
5: three AI SDK components (LangChain core, LangChain, Hugging Face transformers) and two prompt templates
Findings
5, all medium
Tier
High-risk
Obligations checked
28 provider obligations
Stack detected
LangChain · Hugging Face transformers · prompt templates

Context

A toolkit that other teams build on carries a specific risk.

Its examples can be copied into clinical software, and its prompts can end up in front of patients and clinicians. The EU AI Act treats many AI systems in healthcare as high-risk, and the duties depend on what each application is for.

Teams building on a healthcare toolkit ask four questions:

  • What is the toolkit for, and what must it never be used for?
  • Who is responsible for each application built on it?
  • Which prompts and models are in use, and who owns them?
  • What evidence exists for each release?

What we audited

Our audit tool reads the code and the documentation.

It lists the AI components and prompts, places the use in an EU AI Act tier, and checks the evidence against a register of provider obligations. Each finding is tied to an article or control and given a severity.

  • AI inventory

    The SDKs, models and prompt templates in the code and the examples.

  • Risk tier

    Whether the use is high-risk, and what decides it.

  • Obligations

    The 28 provider obligations in the register, checked against the code, the documentation and the logging the toolkit provides.

  • Next steps

    Items that need a running system or an interview, such as incident handling and how teams use the examples.

What the audit found

  • MEDIUMCookbook example

    AI SDK detected: LangChain core

    Why it mattersThe toolkit's prompt chain depends on this library. Changes to it change the outputs.

  • MEDIUMDischarge summary example

    AI SDK detected: Hugging Face transformers

    Why it mattersSummaries are generated by a model. The model version and data handling must be recorded.

  • MEDIUMConfiguration reference

    AI SDK detected: LangChain

    Why it mattersConfiguration sets which model and chain run. It should be versioned.

  • MEDIUMQuestion-and-answer example

    Prompt template detected: inline system prompt

    Why it mattersA prompt in the documentation is copied into applications. Its owner and version must be clear.

  • MEDIUMClinical data mapping configuration

    Prompt template detected: system mapping

    Why it mattersMappings decide how clinical data is read. They need an owner and a review.

  • TIERAudit summary

    Risk tier: high-risk

    Why it mattersDepends on intended use. Brings duties on risk management, documentation, records and oversight.

Across the 28 obligations, the review found evidence in full for 1, in part for 12 and none for 4. 7 need a runtime review, and 4 do not apply.

Some strengths were recorded as well. The toolkit states clearly what data it does and does not send, it includes authentication and encrypted connections, and its developer documentation is extensive.

What we did next

After the report, our forward-deployed engineers either fixed the issues hands-on or advised and worked alongside the team to close them.

The team that builds on the toolkit makes the decisions about its use. Our engineers worked on the parts that are code and configuration, and advised on the parts that are policy.

  • Fixed hands-on

    Our engineers moved the prompts and the clinical mapping into reviewed, versioned configuration with a named owner. They added AI event logging next to the existing web request log, so each model call records its input reference, output reference and version.

  • Advised and worked alongside the team

    Our engineers helped the team write down the intended use of each application, and to record whether the output advises a clinician or decides something. The team owns that decision. They also set up the risk management file, the technical documentation set and a route for serious incidents, with clinical safety and legal staff.

From audit report to technical file A flow from the audit report, with findings ranked by severity, to a triage step the team and an engineer share. From triage the flow splits: engineers fix some findings hands-on, and for others they advise the team, which keeps the decisions on wording and legal basis. Both paths lead to a re-audit of the same system, and the re-audit report goes into the technical file. The two human steps are highlighted. FROM AUDIT TO REMEDIATION AUDITREPORTFindings withseverity and evidence HUMANTRIAGE BYSEVERITYTeam and engineeragree the order FIXHANDS-ONEngineers changecode and config HUMANADVISETHE TEAMTeam decides wordingand legal basis RE-AUDITSAME SYSTEMSame tool, rerunson the release TECHNICALFILERe-audit reportadded to the file Ranked Code fix Advise Checked From audit report to technical file A flow from the audit report, with findings ranked by severity, to a triage step the team and an engineer share. Engineers then fix some findings hands-on, and the team decides the wording and legal basis for the others. The findings are re-audited on the same system, and the re-audit report goes into the technical file. The two human steps are highlighted. AUDIT TOOLAUDIT REPORTFindings with severity and evidence TEAM AND ENGINEER (HUMAN)TRIAGE BY SEVERITYOrder agreed before any change ENGINEERSFIX HANDS-ONModel version, retention, deletion TEAM (HUMAN)ADVISE THE TEAMTeam decides wording and legal basis AUDIT TOOLRE-AUDITSame tool, same system TECHNICAL FILERE-AUDIT REPORT FILEDAdded to the file for the release Ranked by severity Code fix Or advise the team Both paths Checked
Each finding is triaged with the team, fixed or advised on, and re-audited before it goes into the technical file.

What we would advise a team building on this toolkit

  • Write the intended use of each application before any build starts. The tier follows from it.
  • Record whether the output advises a clinician or decides something.
  • Version every prompt and mapping, and give each one an owner.
  • Log AI events, not only web requests.
  • Keep a risk file and a technical file for each release, and a route for serious incidents.

What the team now has

  • A written intended use.

    Each application states what the output is for and who uses it.

  • Prompts and mappings under control.

    Each one has a version and an owner.

  • AI event logs.

    Each model call can be traced to its input reference, output reference and version.

  • A risk file and a technical file.

    Both are set up for each release, and a person signs them off.

  • A route for serious incidents.

    Reports go to a named owner, with corrective action recorded.

How it is built

The toolkit is a library. Teams import it, connect it to records, and build an application on top. The diagram below shows the design the audit pointed towards. The team that builds on the toolkit owns the application and its tier.

From intended use to the tier that applies A flow from the toolkit, a library, to a downstream application built by the adopting team. The team writes down the intended use, then decides whether the output advises a clinician or decides something. The intended use maps to the Annex points that apply, and a person records the tier with its reason. The tier sets the obligations: a risk and prompt file set, AI event logs, a technical file for each release and a route for serious incidents. A person re-checks the tier at each release. Human steps are highlighted. INTENDED USE TO TIER TOOLKITLibrary, examples,prompts DOWNSTREAMAPPLICATIONBuilt by the adoptingteam HUMANINTENDED USEWhat the output is for HUMANADVISES ORDECIDES?Role of output in care ANNEXMAPPINGAnnex I or III pointsapply HUMANTIERRECORDEDTier and reason, signed TECHNICALFILEAnnex IV set perrelease SERIOUS-INCIDENTROUTEReporting andcorrective action HUMANTIER RE-CHECKEach release RISK AND PROMPTFILESHazards, controls, versions,owner AI EVENTLOGSInput and output refs,version Built on Purpose set Output role Use mapped Tier decided Tier sets duties Evidence Each release From intended use to the tier that applies A flow from the toolkit, a library, to a downstream application built by the adopting team. The team writes down the intended use, then decides whether the output advises a clinician or decides something. The intended use maps to the Annex points that apply, and a person records the tier with its reason. The tier sets the obligations: a risk and prompt file set, AI event logs, a technical file for each release and a route for serious incidents. A person re-checks the tier at each release. Human steps are highlighted. TOOLKITLibrary, examples, prompts DOWNSTREAMAPPLICATIONBuilt by the adopting team HUMANINTENDED USEWhat the output is for HUMANADVISES ORDECIDES?Role of output in care ANNEXMAPPINGAnnex I or III points apply HUMANTIERRECORDEDTier and reason, signed RISK AND PROMPTFILESHazards, controls, versions, owner AI EVENTLOGSInput and output refs, version TECHNICALFILEAnnex IV set per release SERIOUS-INCIDENTROUTEReporting and corrective action HUMANTIER RE-CHECKEach release
The intended use sets the tier. The tier sets the evidence the team must hold for each release.

Questions teams ask us

The audit tool places the toolkit in the high-risk tier. The tier that applies to your application depends on its intended use, which is why we start there.
Opt-in request logging exists. The fixes added logs of AI events, so each model call can be traced to its version.
It depends on whether you develop the application yourself or use it under your authority. We map the build to your role first.

Talk through a system like this

Bring your application, your intended use and your clinical review process. We will run the same audit tool on it and walk through the findings, the fixes and the handover pack with you.

Read on: Credit decision support: what the audit found, and how we fixed it · An applicant tracking system: what the audit found, and how we fixed it

* EU AI Act scope: the audit tool places the toolkit in the high-risk tier. The tier of each application depends on its intended use and its role in care.