  Increased availability of data and accessibility of computational tools in
recent years have created unprecedented opportunities for scientific research
driven by statistical analysis. Inherent limitations of statistics impose
constrains on reliability of conclusions drawn from data but misuse of
statistical methods is a growing concern. Significance, hypothesis testing and
the accompanying P-values are being scrutinized as representing most widely
applied and abused practices. One line of critique is that P-values are
inherently unfit to fulfill their ostensible role as measures of scientific
hypothesis's credibility. It has also been suggested that while P-values may
have their role as summary measures of effect, researchers underappreciate the
degree of randomness in the P-value. High variability of P-values would suggest
that having obtained a small P-value in one study, one is, nevertheless, likely
to obtain a much larger P-value in a similarly powered replication study. Thus,
"replicability of P-value" is itself questionable. To characterize P-value
variability one can use prediction intervals whose endpoints reflect the likely
spread of P-values that could have been obtained by a replication study.
Unfortunately, the intervals currently in use, the P-intervals, are based on
unrealistic implicit assumptions. Namely, P-intervals are constructed with the
assumptions that imply substantial chances of encountering large values of
effect size in an observational study, which leads to bias. As an alternative
to P-intervals, we develop a method that gives researchers flexibility by
providing them with the means to control these assumptions. Unlike endpoints of
P-intervals, endpoints of our intervals are directly interpreted as
probabilistic bounds for replication P-values and are resistant to selection
bias contingent upon approximate prior knowledge of the effect size
distribution.
