  With the growing application of AI-based systems in our lives and society,
there is a rising need to ensure that AI-based systems are developed and used
in a responsible way. Fairness is one of the socio-technical concerns that must
be addressed in AI-based systems for this purpose. Unfair AI-based systems,
particularly, unfair AI-based mobile apps, can pose difficulties for a
significant proportion of the global populace. This paper aims to deeply
analyze fairness concerns in AI-based app reviews. We first manually
constructed a ground-truth dataset including a statistical sample of fairness
and non-fairness reviews. Leveraging the ground-truth dataset, we then
developed and evaluated a set of machine learning and deep learning classifiers
that distinguish fairness reviews from non-fairness reviews. Our experiments
show that our best-performing classifier can detect fairness reviews with a
precision of 94%. We then applied the best-performing classifier on
approximately 9.5M reviews collected from 108 AI-based apps and identified
around 92K fairness reviews. While the fairness reviews appear in 23 app
categories, we found that the 'communication' and 'social' app categories have
the highest percentage of fairness reviews. Next, applying the K-means
clustering technique to the 92K fairness reviews, followed by manual analysis,
led to the identification of six distinct types of fairness concerns (e.g.,
'receiving different quality of features and services in different platforms
and devices' and 'lack of transparency and fairness in dealing with
user-generated content'). Finally, the manual analysis of 2,248 app owners'
responses to the fairness reviews identified six root causes (e.g., 'copyright
issues', 'external factors', 'development cost') that app owners report to
justify fairness concerns.
