  In recent years, neural models have been repeatedly touted to exhibit
state-of-the-art performance in recommendation. Nevertheless, multiple recent
studies have revealed that the reported state-of-the-art results of many neural
recommendation models cannot be reliably replicated. A primary reason is that
existing evaluations are performed under various inconsistent protocols.
Correspondingly, these replicability issues make it difficult to understand how
much benefit we can actually gain from these neural models. It then becomes
clear that a fair and comprehensive performance comparison between traditional
and neural models is needed.
  Motivated by these issues, we perform a large-scale, systematic study to
compare recent neural recommendation models against traditional ones in top-n
recommendation from implicit data. We propose a set of evaluation strategies
for measuring memorization performance, generalization performance, and
subgroup-specific performance of recommendation models. We conduct extensive
experiments with 13 popular recommendation models (including two neural models
and 11 traditional ones as baselines) on nine commonly used datasets. Our
experiments demonstrate that even with extensive hyper-parameter searches,
neural models do not dominate traditional models in all aspects, e.g., they
fare worse in terms of average HitRate. We further find that there are areas
where neural models seem to outperform non-neural models, for example, in
recommendation diversity and robustness between different subgroups of users
and items. Our work illuminates the relative advantages and disadvantages of
neural models in recommendation and is therefore an important step towards
building better recommender systems.
