  We consider the issue of biases in scholarly research, specifically, in peer
review. There is a long standing debate on whether exposing author identities
to reviewers induces biases against certain groups, and our focus is on
designing tests to detect the presence of such biases. Our starting point is a
remarkable recent work by Tomkins, Zhang and Heavlin which conducted a
controlled, large-scale experiment to investigate existence of biases in the
peer reviewing of the WSDM conference. We present two sets of results in this
paper. The first set of results is negative, and pertains to the statistical
tests and the experimental setup used in the work of Tomkins et al. We show
that the test employed therein does not guarantee control over false alarm
probability and under correlations between relevant variables coupled with any
of the following conditions, with high probability, can declare a presence of
bias when it is in fact absent: (a) measurement error, (b) model mismatch, (c)
reviewer calibration. Moreover, we show that the setup of their experiment may
itself inflate false alarm probability if (d) bidding is performed in non-blind
manner or (e) popular reviewer assignment procedure is employed. Our second set
of results is positive and is built around a novel approach to testing for
biases that we propose. We present a general framework for testing for biases
in (single vs. double blind) peer review. We then design hypothesis tests that
under minimal assumptions guarantee control over false alarm probability and
non-trivial power even under conditions (a)--(c) as well as propose an
alternative experimental setup which mitigates issues (d) and (e). Finally, we
show that no statistical test can improve over the non-parametric tests we
consider in terms of the assumptions required to control for the false alarm
probability.
