  There is an urgent need for streamlining radiology Quality Assurance (QA)
programs to make them better and faster. Here, we present a novel approach,
Artificial Intelligence (AI)-Based QUality Assurance by Restricted
Investigation of Unequal Scores (AQUARIUS), for re-defining radiology QA, which
reduces human effort by up to several orders of magnitude over existing
approaches. AQUARIUS typically includes automatic comparison of AI-based image
analysis with natural language processing (NLP) on radiology reports. Only the
usually small subset of cases with discordant reads is subsequently reviewed by
human experts. To demonstrate the clinical applicability of AQUARIUS, we
performed a clinical QA study on Intracranial Hemorrhage (ICH) detection in
1936 head CT scans from a large academic hospital. Immediately following image
acquisition, scans were automatically analyzed for ICH using a commercially
available software (Aidoc, Tel Aviv, Israel). Cases rated positive for ICH by
AI (ICH-AI+) were automatically flagged in radiologists' reading worklists,
where flagging was randomly switched off with probability 50%. Using AQUARIUS
with NLP on final radiology reports and targeted expert neuroradiology review
of only 29 discordantly classified cases reduced the human QA effort by 98.5%,
where we found a total of six non-reported true ICH+ cases, with radiologists'
missed ICH detection rates of 0.52% and 2.5% for flagged and non-flagged cases,
respectively. We conclude that AQUARIUS, by combining AI-based image analysis
with NLP-based pre-selection of cases for targeted human expert review, can
efficiently identify missed findings in radiology studies and significantly
expedite radiology QA programs in a hybrid human-machine interoperability
approach.
