  User interaction data is an important source of supervision in counterfactual
learning to rank (CLTR). Such data suffers from presentation bias. Much work in
unbiased learning to rank (ULTR) focuses on position bias, i.e., items at
higher ranks are more likely to be examined and clicked. Inter-item
dependencies also influence examination probabilities, with outlier items in a
ranking as an important example. Outliers are defined as items that observably
deviate from the rest and therefore stand out in the ranking. In this paper, we
identify and introduce the bias brought about by outlier items: users tend to
click more on outlier items and their close neighbors. To this end, we first
conduct a controlled experiment to study the effect of outliers on user clicks.
Next, to examine whether the findings from our controlled experiment generalize
to naturalistic situations, we explore real-world click logs from an e-commerce
platform. We show that, in both scenarios, users tend to click significantly
more on outlier items than on non-outlier items in the same rankings. We show
that this tendency holds for all positions, i.e., for any specific position, an
item receives more interactions when presented as an outlier as opposed to a
non-outlier item. We conclude from our analysis that the effect of outliers on
clicks is a type of bias that should be addressed in ULTR. We therefore propose
an outlier-aware click model that accounts for both outlier and position bias,
called outlier-aware position-based model ( OPBM). We estimate click
propensities based on OPBM ; through extensive experiments performed on both
real-world e-commerce data and semi-synthetic data, we verify the effectiveness
of our outlier-aware click model. Our results show the superiority of OPBM
against baselines in terms of ranking performance and true relevance
estimation.
