  Over the past decade, the field of forensic science has received
recommendations from the National Research Council of the U.S. National Academy
of Sciences, the U.S. National Institute of Standards and Technology, and the
U.S. President's Council of Advisors on Science and Technology to study the
validity and reliability of forensic analyses. More specifically, these
committees recommend estimation of the rates of occurrence of erroneous
conclusions drawn from forensic analyses. "Black box" studies for the various
subjective feature-based comparison methods are intended for this purpose.
  In general, "black box" studies often have unbalanced designs, comparisons
that are not independent, and missing data. These aspects pose difficulty in
the analysis of the results and are often ignored. Instead, interpretation of
the data relies on methods that assume independence between observations and a
balanced experiment. Furthermore, all of these projects are interpreted within
the frequentist framework and result in point estimates associated with
confidence intervals that are confusing to communicate and understand.
  We propose to use an existing likelihood-free Bayesian inference method,
called Approximate Bayesian Computation (ABC), that is capable of handling
unbalanced designs, dependencies among the observations, and missing data. ABC
allows for studying the parameters of interest without recourse to incoherent
and misleading measures of uncertainty such as confidence intervals. By taking
into account information from all decision categories for a given examiner and
information from the population of examiners, our method also allows for
quantifying the risk of error for the given examiner, even when no error has
been recorded for that examiner.
  We illustrate our proposed method by reanalysing the results of the "Noblis
Black Box" study by Ulery et al. in 2011.
