  We investigated the use of equivalent relative utility (ERU) to evaluate the
effectiveness of artificial intelligence (AI)-enabled rule-out devices that use
AI to identify and autonomously remove non-cancer patient images from
radiologist review in screening mammography.We reviewed two performance metrics
that can be used to compare the diagnostic performance between the
radiologist-with-rule-out-device and radiologist-without-device workflows:
positive/negative predictive values (PPV/NPV) and equivalent relative utility
(ERU). To demonstrate the use of the two evaluation metrics, we applied both
methods to a recent US-based study that reported an improved performance of the
radiologist-with-device workflow compared to the one without the device by
retrospectively applying their AI algorithm to a large mammography dataset. We
further applied the ERU method to a European study utilizing their reported
recall rates and cancer detection rates at different thresholds of their AI
algorithm to compare the potential utility among different thresholds. For the
study using US data, neither the PPV/NPV nor the ERU method can conclude a
significant improvement in diagnostic performance for any of the algorithm
thresholds reported. For the study using European data, ERU values at lower AI
thresholds are found to be higher than that at a higher threshold because more
false-negative cases would be ruled-out at higher threshold, reducing the
overall diagnostic performance. Both PPV/NPV and ERU methods can be used to
compare the diagnostic performance between the radiologist-with-device workflow
and that without. One limitation of the ERU method is the need to measure the
baseline, standard-of-care relative utility (RU) value for mammography
screening in the US. Once the baseline value is known, the ERU method can be
applied to large US datasets without knowing the true prevalence of the
dataset.
