  In this work, we collect data from runs of Krylov subspace methods and
pipelined Krylov algorithms in an effort to understand and model the impact of
machine noise and other sources of variability on performance. We find large
variability of Krylov iterations between compute nodes for standard methods
that is reduced in pipelined algorithms, directly supporting conjecture, as
well as large variation between statistical distributions of runtimes across
iterations. Based on these results, we improve upon a previously introduced
nondeterministic performance model by allowing iterations to fluctuate over
time. We present our data from runs of various Krylov algorithms across
multiple platforms as well as our updated non-stationary model that provides
good agreement with observations. We also suggest how it can be used as a
predictive tool.
