  Measuring performance & quantifying a performance change are core evaluation
techniques in programming language and systems research. Of 122 recent
scientific papers, as many as 65 included experimental evaluation that
quantified a performance change using a ratio of execution times. Few of these
papers evaluated their results with the level of rigour that has come to be
expected in other experimental sciences. The uncertainty of measured results
was largely ignored. Scarcely any of the papers mentioned uncertainty in the
ratio of the mean execution times, and most did not even mention uncertainty in
the two means themselves. Most of the papers failed to address the
non-deterministic execution of computer programs (caused by factors such as
memory placement, for example), and none addressed non-deterministic
compilation. It turns out that the statistical methods presented in the
computer systems performance evaluation literature for the design and summary
of experiments do not readily allow this either. This poses a hazard to the
repeatability, reproducibility and even validity of quantitative results.
  Inspired by statistical methods used in other fields of science, and building
on results in statistics that did not make it to introductory textbooks, we
present a statistical model that allows us both to quantify uncertainty in the
ratio of (execution time) means and to design experiments with a rigorous
treatment of those multiple sources of non-determinism that might impact
measured performance. Better still, under our framework summaries can be as
simple as "system A is faster than system B by 5.5% $\pm$ 2.5%, with 95%
confidence", a more natural statement than those derived from typical current
practice, which are often misinterpreted.
  November 2013
