  Online experiments are ubiquitous. As the scale of experiments has grown, so
has the complexity of their design and implementation. In response, firms have
developed software frameworks for designing and deploying online experiments.
Ensuring that experiments in these frameworks are correctly designed and that
their results are trustworthy---referred to as *internal validity*---can be
difficult. Currently, verifying internal validity requires manual inspection by
someone with substantial expertise in experimental design.
  We present the first approach for statically checking the internal validity
of online experiments. Our checks are based on well-known problems that arise
in experimental design and causal inference. Our analyses target PlanOut, a
widely deployed, open-source experimentation framework that uses a
domain-specific language to specify and run complex experiments. We have built
a tool, PlanAlyzer, that checks PlanOut programs for a variety of threats to
internal validity, including failures of randomization, treatment assignment,
and causal sufficiency. PlanAlyzer uses its analyses to automatically generate
*contrasts*, a key type of information required to perform valid statistical
analyses over experimental results. We demonstrate PlanAlyzer's utility on a
corpus of PlanOut scripts deployed in production at Facebook, and we evaluate
its ability to identify threats to validity on a mutated subset of this corpus.
PlanAlyzer has both precision and recall of 92% on the mutated corpus, and 82%
of the contrasts it automatically generates match hand-specified data.
