  Despite the success of deep neural networks (DNNs) in enabling on-device
voice assistants, increasing evidence of bias and discrimination in machine
learning is raising the urgency of investigating the fairness of these systems.
Speaker verification is a form of biometric identification that gives access to
voice assistants. Due to a lack of fairness metrics and evaluation frameworks
that are appropriate for testing the fairness of speaker verification
components, little is known about how model performance varies across
subgroups, and what factors influence performance variation. To tackle this
emerging challenge, we design and develop SVEva Fair, an accessible, actionable
and model-agnostic framework for evaluating the fairness of speaker
verification components. The framework provides evaluation measures and
visualisations to interrogate model performance across speaker subgroups and
compare fairness between models. We demonstrate SVEva Fair in a case study with
end-to-end DNNs trained on the VoxCeleb datasets to reveal potential bias in
existing embedded speech recognition systems based on the demographic
attributes of speakers. Our evaluation shows that publicly accessible benchmark
models are not fair and consistently produce worse predictions for some
nationalities, and for female speakers of most nationalities. To pave the way
for fair and reliable embedded speaker verification, SVEva Fair has been
implemented as an open-source python library and can be integrated into the
embedded ML development pipeline to facilitate developers and researchers in
troubleshooting unreliable speaker verification performance, and selecting high
impact approaches for mitigating fairness challenges
