  When machine-learning algorithms are used in high-stakes decisions, we want
to ensure that their deployment leads to fair and equitable outcomes. This
concern has motivated a fast-growing literature that focuses on diagnosing and
addressing disparities in machine predictions. However, many machine
predictions are deployed to assist in decisions where a human decision-maker
retains the ultimate decision authority. In this article, we therefore consider
in a formal model and in a lab experiment how properties of machine predictions
affect the resulting human decisions. In our formal model of statistical
decision-making, we show that the inclusion of a biased human decision-maker
can revert common relationships between the structure of the algorithm and the
qualities of resulting decisions. Specifically, we document that excluding
information about protected groups from the prediction may fail to reduce, and
may even increase, ultimate disparities. In the lab experiment, we demonstrate
how predictions informed by gender-specific information can reduce average
gender disparities in decisions. While our concrete theoretical results rely on
specific assumptions about the data, algorithm, and decision-maker, and the
experiment focuses on a particular prediction task, our findings show more
broadly that any study of critical properties of complex decision systems, such
as the fairness of machine-assisted human decisions, should go beyond focusing
on the underlying algorithmic predictions in isolation.
