  Online experimentation platforms abstract away many of the details of
experimental design, ensuring experimenters do not have to worry about
sampling, randomisation, subject tracking, data collection, metric definition
and interpretation of results. The recent success and rapid adoption of these
platforms in the industry might in part be attributed to the ease-of-use these
abstractions provide. Previous authors have pointed out there are common
pitfalls to avoid when running controlled experiments on the web and emphasised
the need for experts familiar with the entire software stack to be involved in
the process.
  In this paper, we argue that these pitfalls and the need to understand the
underlying complexity are not the result of shortcomings specific to existing
platforms which might be solved by better platform design. We postulate that
they are a direct consequence of what is commonly referred to as "the law of
leaky abstractions". That is, it is an inherent feature of any software
platform that details of its implementation leak to the surface, and that in
certain situations, the platform's consumers necessarily need to understand
details of underlying systems in order to make proficient use of it.
  We present several examples of this concept, including examples from
literature, and suggest some possible mitigation strategies that can be
employed to reduce the impact of abstraction leakage. The conceptual framework
put forward in this paper allows us to explicitly categorize experimentation
pitfalls in terms of which specific abstraction is leaking, thereby aiding
implementers and users of these platforms to better understand and tackle the
challenges they face.
