  Simulation can enable the study of recommender system (RS) evolution while
circumventing many of the issues of empirical longitudinal studies; simulations
are comparatively easier to implement, are highly controlled, and pose no
ethical risk to human participants. How simulation can best contribute to
scientific insight about RS alongside qualitative and quantitative empirical
approaches is an open question. Philosophers and researchers have long debated
the epistemological nature of simulation compared to wholly theoretical or
empirical methods. Simulation is often implicitly or explicitly conceptualized
as occupying a middle ground between empirical and theoretical approaches,
allowing researchers to realize the benefits of both. However, what is often
ignored in such arguments is that without firm grounding in any single
methodological tradition, simulation studies have no agreed upon scientific
norms or standards, resulting in a patchwork of theoretical motivations,
approaches, and implementations that are difficult to reconcile. In this
position paper, we argue that simulation studies of RS are conceptually similar
to empirical experimental approaches and therefore can be evaluated using the
standards of empirical research methods. Using this empirical lens, we argue
that the combination of high heterogeneity in approaches and low transparency
in methods in simulation studies of RS has limited their interpretability,
generalizability, and replicability. We contend that by adopting standards and
practices common in empirical disciplines, simulation researchers can mitigate
many of these weaknesses.
