  Successful HPC software applications are long-lived. When ported across
machines and their compilers, these applications often produce different
numerical results, many of which are unacceptable. Such variability is also a
concern while optimizing the code more aggressively to gain performance.
Efficient tools that help locate the program units (files and functions) within
which most of the variability occurs are badly needed, both to plan for code
ports and to root-cause errors due to variability when they happen in the
field. In this work, we offer an enhanced version of the open-source testing
framework FLiT to serve these roles. Key new features of FLiT include a suite
of bisection algorithms that help locate the root causes of variability.
Another added feature allows an analysis of the tradeoffs between performance
and the degree of variability. Our new contributions also include a collection
of case studies. Results on the MFEM finite-element library include
variability/performance tradeoffs, and the identification of a (hitherto
unknown) abnormal level of result-variability even under mild compiler
optimizations. Results from studying the Laghos proxy application include
identifying a significantly divergent floating-point result-variability and
successful root-causing down to the problematic function over as little as 14
program executions. Finally, in an evaluation of 4,376 controlled injections of
floating-point perturbations on the LULESH proxy application, we showed that
the FLiT framework has 100 precision and recall in discovering the file and
function locations of the injections all within an average of only 15 program
executions.
