  Recent advances in machine learning have led to growing interest in
Explainable AI (xAI) to enable humans to gain insight into the decision-making
of machine learning models. Despite this recent interest, the utility of xAI
techniques has not yet been characterized in human-machine teaming.
Importantly, xAI offers the promise of enhancing team situational awareness
(SA) and shared mental model development, which are the key characteristics of
effective human-machine teams. Rapidly developing such mental models is
especially critical in ad hoc human-machine teaming, where agents do not have a
priori knowledge of others' decision-making strategies. In this paper, we
present two novel human-subject experiments quantifying the benefits of
deploying xAI techniques within a human-machine teaming scenario. First, we
show that xAI techniques can support SA ($p<0.05)$. Second, we examine how
different SA levels induced via a collaborative AI policy abstraction affect ad
hoc human-machine teaming performance. Importantly, we find that the benefits
of xAI are not universal, as there is a strong dependence on the composition of
the human-machine team. Novices benefit from xAI providing increased SA
($p<0.05$) but are susceptible to cognitive overhead ($p<0.05$). On the other
hand, expert performance degrades with the addition of xAI-based support
($p<0.05$), indicating that the cost of paying attention to the xAI outweighs
the benefits obtained from being provided additional information to enhance SA.
Our results demonstrate that researchers must deliberately design and deploy
the right xAI techniques in the right scenario by carefully considering
human-machine team composition and how the xAI method augments SA.
