  In recent years, machine learning algorithms have become ubiquitous in a
multitude of high-stakes decision-making applications. The unparalleled ability
of machine learning algorithms to learn patterns from data also enables them to
incorporate biases embedded within. A biased model can then make decisions that
disproportionately harm certain groups in society -- limiting their access to
financial services, for example. The awareness of this problem has given rise
to the field of Fair ML, which focuses on studying, measuring, and mitigating
unfairness in algorithmic prediction, with respect to a set of protected groups
(e.g., race or gender). However, the underlying causes for algorithmic
unfairness still remain elusive, with researchers divided between blaming
either the ML algorithms or the data they are trained on. In this work, we
maintain that algorithmic unfairness stems from interactions between models and
biases in the data, rather than from isolated contributions of either of them.
To this end, we propose a taxonomy to characterize data bias and we study a set
of hypotheses regarding the fairness-accuracy trade-offs that fairness-blind ML
algorithms exhibit under different data bias settings. On our real-world
account-opening fraud use case, we find that each setting entails specific
trade-offs, affecting fairness in expected value and variance -- the latter
often going unnoticed. Moreover, we show how algorithms compare differently in
terms of accuracy and fairness, depending on the biases affecting the data.
Finally, we note that under specific data bias conditions, simple
pre-processing interventions can successfully balance group-wise error rates,
while the same techniques fail in more complex settings.
