  Disaggregated performance metrics across demographic groups are a hallmark of
fairness assessments in computer vision. These metrics successfully
incentivized performance improvements on person-centric tasks such as face
analysis and are used to understand risks of modern models. However, there is a
lack of discussion on the vulnerabilities of these measurements for more
complex computer vision tasks. In this paper, we consider multi-label image
classification and, specifically, object categorization tasks. First, we
highlight design choices and trade-offs for measurement that involve more
nuance than discussed in prior computer vision literature. These challenges are
related to the necessary scale of data, definition of groups for images, choice
of metric, and dataset imbalances. Next, through two case studies using modern
vision models, we demonstrate that naive implementations of these assessments
are brittle. We identify several design choices that look merely like
implementation details but significantly impact the conclusions of assessments,
both in terms of magnitude and direction (on which group the classifiers work
best) of disparities. Based on ablation studies, we propose some
recommendations to increase the reliability of these assessments. Finally,
through a qualitative analysis we find that concepts with large disparities
tend to have varying definitions and representations between groups, with
inconsistencies across datasets and annotators. While this result suggests
avenues for mitigation through more consistent data collection, it also
highlights that ambiguous label definitions remain a challenge when performing
model assessments. Vision models are expanding and becoming more ubiquitous; it
is even more important that our disparity assessments accurately reflect the
true performance of models.
