  Online marketplace designers frequently run A/B tests to measure the impact
of proposed product changes. However, given that marketplaces are inherently
connected, total average treatment effect estimates obtained through Bernoulli
randomized experiments are often biased due to violations of the stable unit
treatment value assumption. This can be particularly problematic for
experiments that impact sellers' strategic choices, affect buyers' preferences
over items in their consideration set, or change buyers' consideration sets
altogether. In this work, we measure and reduce bias due to interference in
online marketplace experiments by using observational data to create clusters
of similar listings, and then using those clusters to conduct
cluster-randomized field experiments. We provide a lower bound on the magnitude
of bias due to interference by conducting a meta-experiment that randomizes
over two experiment designs: one Bernoulli randomized, one cluster randomized.
In both meta-experiment arms, treatment sellers are subject to a different
platform fee policy than control sellers, resulting in different prices for
buyers. By conducting a joint analysis of the two meta-experiment arms, we find
a large and statistically significant difference between the total average
treatment effect estimates obtained with the two designs, and estimate that
32.60% of the Bernoulli-randomized treatment effect estimate is due to
interference bias. We also find weak evidence that the magnitude and/or
direction of interference bias depends on extent to which a marketplace is
supply- or demand-constrained, and analyze a second meta-experiment to
highlight the difficulty of detecting interference bias when treatment
interventions require intention-to-treat analysis.
