Diagnostic AI Evaluation Checklist
Purpose
Diagnostic AI claims live or die on evaluation design, and the recurring failures are known: patient leakage across train-test splits, retrospective archives standing in for prospective performance, subgroup performance hidden inside averages, and an intended use quietly wider than anything tested. This skill reviews an evaluation plan and drafts the checklist that names those gaps, mapped onto the AI-specific reporting standards. It reviews plans; it approves nothing.
The five review layers
- Dataset and splits. The first question is unit of splitting: patient-level, not examination-level, or the same patient's earlier images teach the model their later labels. Then provenance: inclusion criteria, missingness, and the gap between the archive and the population the intended use names.
- Validation. Internal performance on a held-out split is development evidence. External validation (other site, other scanner generation, other population) is the layer that supports a deployment claim. Where the plan claims emergency department use and validates on a retrospective archive, the gap between tested and claimed use is the central finding.
- Subgroups and shift. Averages hide subgroup performance; the checklist asks for performance by age, sex, device, and clinically relevant subgroups, with intervals. Dataset shift (scanner replacement, protocol changes, referral pattern drift) is a deployment risk that no single validation fully retires, which is why monitoring plans appear here.
- Human factors. Where the tool changes clinician behavior (triage prioritization, alert gating), DECIDE-AI's live-evaluation concerns apply: who sees the output, what they do with it, and how automation bias will be detected.
- Reporting. STARD-AI and QUADAS-AI for accuracy studies involving AI, TRIPOD+AI for model development reporting, CONSORT-AI and SPIRIT-AI for trials of AI interventions. The checklist asks for the intended use and the tested-use boundary to be stated in the same sentence.
Common failure modes
- Examination-level splits presented as patient-level validation.
- Retrospective single-center evidence offered for a multi-site deployment claim.
- Subgroup analysis promised, then reported as an overall AUC.
- Evaluation of the model in isolation, with no measurement of what clinicians do with its output.
Integration with other skills
The Healthcare AI Safety Checklist covers the governance side for the same tool, the Diagnostic Accuracy Study Planner covers the non-AI accuracy design it extends, and the Risk of Bias Assistant applies QUADAS-3 and QUADAS-3-style appraisal to published studies the plan cites.