  We propose a novel approach to the statistical analysis of stochastic
simulation models and, especially, agent-based models (ABMs). Our main goal is
to provide fully automated, model-independent and tool-supported techniques and
algorithms to inspect simulations and perform counterfactual analysis. Our
approach: (i) is easy-to-use by the modeller, (ii) improves reproducibility of
results, (iii) optimizes running time given the modeller's machine, (iv)
automatically chooses the number of required simulations and simulation steps
to reach user-specified statistical confidence, and (v) automates a variety of
statistical tests. In particular, our techniques are designed to distinguish
the transient dynamics of the model from its steady-state behaviour (if any),
estimate properties in both 'phases', and provide indications on the
(non-)ergodic nature of the simulated processes - which, in turn, allows one to
gauge the reliability of a steady-state analysis. Estimates are equipped with
statistical guarantees, allowing for robust comparisons across computational
experiments. To demonstrate the effectiveness of our approach, we apply it to
two models from the literature: a large-scale macro-financial ABM and a small
scale prediction market model. Compared to prior analyses of these models, we
obtain new insights and we are able to identify and fix some erroneous
conclusions.
