  Accurate performance estimation of future many-node machines is challenging
because it requires detailed simulation models of both node and network.
However, simulating the full system in detail is unfeasible in terms of compute
and memory resources. State-of-the-art techniques use a two-phase approach that
combines detailed simulation of a single node with network-only simulation of
the full system. We show that these techniques, where the detailed node
simulation is done in isolation, are inaccurate because they ignore two
important node-level effects: compute time variability, and inter-node
communication.
  We propose a novel three-stage simulation method to allow scalable and
accurate many-node simulation, combining native profiling, detailed node
simulation and high-level network simulation. By including timing variability
and the impact of external nodes, our method leads to more accurate estimates.
We validate our technique against measurements on a multi-node cluster, and
report an average 6.7% error on 64 nodes (maximum error of 12%), compared to on
average 27% error and up to 54% when timing variability and the scaling
overhead are ignored. At higher node counts, the prediction error of ignoring
variable timings and scaling overhead continues to increase compared to our
technique, and may lead to selecting the wrong optimal cluster configuration.
  Using our technique, we are able to accurately project performance to
thousands of nodes within a day of simulation time, using only a single or a
few simulation hosts. Our method can be used to quickly explore large many-node
design spaces, including node micro-architecture, node count and network
configuration.
