  A fundamental challenge of software testing is the statistically
well-grounded extrapolation from program behaviors observed during testing. For
instance, a security researcher who has run the fuzzer for a week has currently
no means (i) to estimate the total number of feasible program branches, given
that only a fraction has been covered so far, (ii) to estimate the additional
time required to cover 10% more branches, or (iii) to assess the residual risk
that a vulnerability exists when no vulnerability has been discovered. Failing
to discover a vulnerability, does not mean that none exists---even if the
fuzzer was run for a week (or a year). Hence, testing provides no formal
correctness guarantees.
  In this article, I establish an unexpected connection with the otherwise
unrelated scientific field of ecology, and introduce a statistical framework
that models Software Testing and Analysis as Discovery of Species (STADS). For
instance, in order to study the species diversity of arthropods in a tropical
rain forest, ecologists would first sample a large number of individuals from
that forest, determine their species, and extrapolate from the properties
observed in the sample to properties of the whole forest. The estimation (i) of
the total number of species, (ii) of the additional sampling effort required to
discover 10% more species, or (iii) of the probability to discover a new
species are classical problems in ecology. The STADS framework draws from over
three decades of research in ecological biostatistics to address the
fundamental extrapolation challenge for automated test generation. Our
preliminary empirical study demonstrates a good estimator performance even for
a fuzzer with adaptive sampling bias---AFL, a state-of-the-art vulnerability
detection tool. The STADS framework provides statistical correctness guarantees
with quantifiable accuracy.
