  The field of transparent Machine Learning (ML) has contributed many novel
methods aiming at better interpretability for computer vision and ML models in
general. But how useful the explanations provided by transparent ML methods are
for humans remains difficult to assess. Most studies evaluate interpretability
in qualitative comparisons, they use experimental paradigms that do not allow
for direct comparisons amongst methods or they report only offline experiments
with no humans in the loop. While there are clear advantages of evaluations
with no humans in the loop, such as scalability, reproducibility and less
algorithmic bias than with humans in the loop, these metrics are limited in
their usefulness if we do not understand how they relate to other metrics that
take human cognition into account. Here we investigate the quality of
interpretable computer vision algorithms using techniques from psychophysics.
In crowdsourced annotation tasks we study the impact of different
interpretability approaches on annotation accuracy and task time. In order to
relate these findings to quality measures for interpretability without humans
in the loop we compare quality metrics with and without humans in the loop. Our
results demonstrate that psychophysical experiments allow for robust quality
assessment of transparency in machine learning. Interestingly the quality
metrics computed without humans in the loop did not provide a consistent
ranking of interpretability methods nor were they representative for how useful
an explanation was for humans. These findings highlight the potential of
methods from classical psychophysics for modern machine learning applications.
We hope that our results provide convincing arguments for evaluating
interpretability in its natural habitat, human-ML interaction, if the goal is
to obtain an authentic assessment of interpretability.
