Healthcare AI Safety Checklist
Purpose
This skill produces a structured governance checklist for a healthcare AI tool. It is designed for teams that need a first, structured pass at safety and governance before they involve lawyers, notified bodies, and regulators. It is a planning document, not a certification, and it never issues an approval.
The regulatory landscape as of August 2026
This skill is intentionally current to the EU AI Act's main applicability date. What a team needs to know in August 2026:
-
The EU AI Act became generally applicable on 2 August 2026. From that date, the AI Office and member state authorities began enforcing the Act, and transparency obligations (for example, disclosing AI-generated content and AI interaction) took effect. Prohibited practices were already banned from February 2025, and obligations on general-purpose AI models applied from August 2025.
-
High-risk obligations are staggered. Under the Digital Omnibus amendments, most high-risk Annex III systems (biometrics, education, employment, and similar) face obligations from 2 December 2027, and systems embedded in regulated products, which includes much medical device AI, face obligations from 2 August 2028. A team building healthcare AI in August 2026 should plan for those dates, not assume the obligations are live today.
-
Medical devices are separately regulated. The EU MDR applies to software that is a medical device, and that determination is separate from the AI Act. The first question for any clinical tool is whether it is a device under MDR or under national device law.
-
The US landscape centers on the FDA's lifecycle approach: the January 2025 final guidance on AI-enabled device software functions, and the December 2024 final guidance on Predetermined Change Control Plans (PCCPs), which let manufacturers plan software updates in advance.
-
CHAI and the assurance ecosystem. The Coalition for Health AI has published a blueprint for trustworthy AI and supporting assurance standards. These are not law, but they are becoming the de facto framework that purchasers and health systems ask for, and CHAI's Applied Model Card format is the artifact they increasingly request from vendors.
-
AI-specific reporting standards. When reviewing a tool's evaluation literature, check reporting against TRIPOD+AI (prediction models), DECIDE-AI (live clinical evaluation of AI systems), and CHART (reporting LLM use in research). Weak reporting is itself a governance signal.
How to use the checklist
Run the skill with a written description of the tool: what it does, who uses it, in what setting, and with what data. The output has ten sections. Work through them with the product, engineering, clinical, and legal stakeholders. Anything the team cannot answer becomes an open question, and any regulatory classification question becomes an open question that only qualified counsel can close.
What the checklist does not do
- It does not determine whether a tool is a medical device.
- It does not classify a system as high risk under the AI Act.
- It does not certify, approve, or clear anything.
- It does not interpret the law. Where the law matters, the output says so and adds an open question.
Worked example: research-facing tool (EU)
For a tool that summarizes clinical trial eligibility criteria for research staff, the central governance point is that the tool must not determine eligibility. The checklist accordingly emphasizes human oversight, a comparison of parsed output with the original protocol, and a regulatory readiness note that distinguishes obligations in force from obligations that start later.
Worked example: public health chatbot (EU)
For a public-facing information chatbot, transparency obligations matter most: users must know they are talking to AI, and content must come from a verified, versioned library. The regulatory readiness note flags the transparency and AI literacy obligations that apply as of 2026, while keeping the high-risk question as an open item to document.
Common failure modes
- Certification language: the checklist says "assess" and "document", never "approved". If the model drifts into approval language, that is a failure.
- Ignoring the timeline: saying "the AI Act applies from August 2026" without noting that high-risk product-embedded obligations start in 2028. Both are true and both belong in the output.
- Regulatory overreach: answering classification questions that require legal review. Those belong in open_questions.
- US/EU conflation: obligations differ sharply; the jurisdiction input exists for this reason.