  A typical benchmark dataset for recommender system (RecSys) evaluation
consists of user-item interactions generated on a platform within a time
period. The interaction generation mechanism partially explains why a user
interacts with (e.g.,like, purchase, rate) an item, and the context of when a
particular interaction happened. In this study, we conduct a meticulous
analysis on the MovieLens dataset and explain the potential impact on using the
dataset for evaluating recommendation algorithms. We make a few main findings
from our analysis. First, there are significant differences in user
interactions at the different stages when a user interacts with the MovieLens
platform. The early interactions largely define the user portrait which affect
the subsequent interactions. Second, user interactions are highly affected by
the candidate movies that are recommended by the platform's internal
recommendation algorithm(s). Removal of interactions that happen nearer to the
last few interactions of a user leads to increasing difficulty in learning user
preference, thus deteriorating recommendation accuracy. Third, changing the
order of user interactions makes it more difficult for sequential algorithms to
capture the progressive interaction process. Based on these findings, we
further discuss the discrepancy between the interaction generation mechanism
that is employed by the MovieLens system and that of typical real world
recommendation scenarios. In summary, models that achieve excellent
recommendation accuracy on the MovieLens dataset may not demonstrate superior
performance in practice for at least two kinds of differences: (i) the
differences in the contexts of user-item interaction generation, and (ii) the
differences in user knowledge about the item collections.
