  Background: Before being used to inform patient care, a risk prediction model
needs to be validated in a representative sample from the target population.
The finite size of the validation sample entails that there is uncertainty with
respect to estimates of model performance. We apply value-of-information
methodology as a framework to quantify the consequence of such uncertainty in
terms of NB. Methods: We define the Expected Value of Perfect Information
(EVPI) for model validation as the expected loss in NB due to not confidently
knowing which of the alternative decisions confers the highest NB at a given
risk threshold. We propose methods for EVPI calculations based on Bayesian or
ordinary bootstrapping of NBs, as well as an asymptotic approach supported by
the central limit theorem. We conducted brief simulation studies to compare the
performance of these methods, and used subsets of data from an international
clinical trial for predicting mortality after myocardial infarction as a case
study. Results: The three computation methods generated similar EVPI values in
simulation studies. In the case study, at the pre-specified threshold of 0.02,
the best decision with current information would be to use the model, with an
expected incremental NB of 0.0020 over treating all. At this threshold, EVPI
was 0.0005 (a relative EVPI of 25%). When scaled to the annual number of heart
attacks in the US, this corresponds to a loss of 400 true positives, or extra
19,600 false positives (unnecessary treatments) per year, indicating the value
of further model validation. As expected, the validation EVPI generally
declined with larger samples. Conclusion: Value-of-information methods can be
applied to the NB calculated during external validation of clinical prediction
models to provide a decision-theoretic perspective to the consequences of
uncertainty.
