  The National Science Foundation (NSF) will be experimenting with a new
distributed approach to reviewing proposals, whereby a group of principal
investigators (PIs) or proposers in a subfield act as reviewers for the
proposals submitted by the same set of PIs. To encourage honesty, PIs' chances
for getting funded are tied to the quality of their reviews (with respect to
the reviews provided by the entire group), in addition to the quality of their
proposals. Intuitively, this approach can more fairly distribute the review
workload, discourage frivolous proposal submission, and encourage high quality
reviews. On the other hand, this method has already raised concerns about the
integrity of the process and the possibility of strategic manipulation. In this
paper, we take a closer look at three specific issues in an attempt to gain a
better understanding of the strengths and limitations of the new process beyond
first impressions and anecdotal evidence. We start by considering the benefits
and drawbacks of bundling the quality of PIs' reviews with the scientific merit
of their proposals. We then consider the issue of collusion and favoritism.
Finally, we examine whether the new process puts controversial proposals at a
disadvantage. We conclude that some benefits of using review quality as an
incentive mechanism may outweigh its drawbacks. On the other hand, even a
coalition of two PIs can cause significant harm to the process, as the built-in
incentives are not strong enough to deter collusion. While we also confirm the
common suspicion that the process is skewed toward non-controversial proposals,
the more unexpected finding is that among equally controversial proposals,
those of lower quality get a leg up through this process. Thus the process not
only favors non-controversial proposals, but in some sense, mediocrity. We also
discuss possible ways to improve this review process.
