  Randomized experiments, or A/B tests are used to estimate the causal impact
of a feature on the behavior of users by creating two parallel universes in
which members are simultaneously assigned to treatment and control. However, in
social network settings, members interact, such that the impact of a feature is
not always contained within the treatment group. Researchers have developed a
number of experimental designs to estimate network effects in social settings.
Alternatively, naturally occurring exogenous variation, or 'natural
experiments,' allow researchers to recover causal estimates of peer effects
from observational data in the absence of experimental manipulation. Natural
experiments trade off the engineering costs and some of the ethical concerns
associated with network randomization with the search costs of finding
situations with natural exogenous variation. To mitigate the search costs
associated with discovering natural counterfactuals, we identify a common
engineering requirement used to scale massive online systems, in which natural
exogenous variation is likely to exist: notification queueing. We identify two
natural experiments on the LinkedIn platform based on the order of notification
queues to estimate the causal impact of a received message on the engagement of
a recipient. We show that receiving a message from another member significantly
increases a member's engagement, but that some popular observational
specifications, such as fixed-effects estimators, overestimate this effect by
as much as 2.7x. We then apply the estimated network effect coefficients to a
large body of past experiments to quantify the extent to which it changes our
interpretation of experimental results. The study points to the benefits of
using messaging queues to discover naturally occurring counterfactuals for the
estimation of causal effects without experimenter intervention.
