  Explanations are hypothesized to improve human understanding of machine
learning models and achieve a variety of desirable outcomes, ranging from model
debugging to enhancing human decision making. However, empirical studies have
found mixed and even negative results. An open question, therefore, is under
what conditions explanations can improve human understanding and in what way.
Using adapted causal diagrams, we provide a formal characterization of the
interplay between machine explanations and human understanding, and show how
human intuitions play a central role in enabling human understanding.
Specifically, we identify three core concepts of interest that cover all
existing quantitative measures of understanding in the context of human-AI
decision making: task decision boundary, model decision boundary, and model
error. Our key result is that without assumptions about task-specific
intuitions, explanations may potentially improve human understanding of model
decision boundary, but they cannot improve human understanding of task decision
boundary or model error. To achieve complementary human-AI performance, we
articulate possible ways on how explanations need to work with human
intuitions. For instance, human intuitions about the relevance of features
(e.g., education is more important than age in predicting a person's income)
can be critical in detecting model error. We validate the importance of human
intuitions in shaping the outcome of machine explanations with empirical
human-subject studies. Overall, our work provides a general framework along
with actionable implications for future algorithmic development and empirical
experiments of machine explanations.
