  In an A/B test, the typical objective is to measure the total average
treatment effect (TATE), which measures the difference between the average
outcome if all users were treated and the average outcome if all users were
untreated. However, a simple difference-in-means estimator will give a biased
estimate of the TATE when outcomes of control units depend on the outcomes of
treatment units, an issue we refer to as test-control interference. Using a
simulation built on top of data from Airbnb, this paper considers the use of
methods from the network interference literature for online marketplace
experimentation. We model the marketplace as a network in which an edge exists
between two sellers if their goods substitute for one another. We then simulate
seller outcomes, specifically considering a "status quo" context and
"treatment" context that forces all sellers to lower their prices. We use the
same simulation framework to approximate TATE distributions produced by using
blocked graph cluster randomization, exposure modeling, and the Hajek estimator
for the difference in means. We find that while blocked graph cluster
randomization reduces the bias of the naive difference-in-means estimator by as
much as 62%, it also significantly increases the variance of the estimator. On
the other hand, the use of more sophisticated estimators produces mixed
results. While some provide (small) additional reductions in bias and small
reductions in variance, others lead to increased bias and variance. Overall,
our results suggest that experiment design and analysis techniques from the
network experimentation literature are promising tools for reducing bias due to
test-control interference in marketplace experiments.
