  We present a novel evaluation framework for representation bias in latent
factor recommendation (LFR) algorithms. Our framework introduces the concept of
attribute association bias in recommendations allowing practitioners to explore
how recommendation systems can introduce or amplify stakeholder representation
harm. Attribute association bias (AAB) occurs when sensitive attributes become
semantically captured or entangled in the trained recommendation latent space.
This bias can result in the recommender reinforcing harmful stereotypes, which
may result in downstream representation harms to system consumer and provider
stakeholders. LFR models are at risk of experiencing AAB due to their ability
to entangle explicit and implicit attributes into the trained latent space.
Understanding this phenomenon is essential due to the increasingly common use
of entity vectors as attributes in downstream components in hybrid industry
recommendation systems. We provide practitioners with a framework for executing
disaggregated evaluations of AAB within broader algorithmic auditing
frameworks. Inspired by research in natural language processing (NLP) observing
gender bias in word embeddings, our framework introduces AAB evaluation methods
specifically for recommendation entity vectors. We present four evaluation
strategies for sensitive AAB in LFR models: attribute bias directions,
attribute association bias metrics, classification for explaining bias, and
latent space visualization. We demonstrate the utility of our framework by
evaluating user gender AAB regarding podcast genres with an industry case study
of a production-level DNN recommendation model. We uncover significant levels
of user gender AAB when user gender is used and removed as a model feature
during training, pointing to the potential for systematic bias in LFR model
outputs.
