  Fault tolerance for the upcoming exascale generation has long been an area of
active research. One of the components of a fault tolerance strategy is
checkpointing. Petascale-level checkpointing is demonstrated through a new
mechanism for virtualization of the InfiniBand UD (unreliable datagram) mode,
and for updating the remote address on each UD-based send, due to lack of a
fixed peer. Note that InfiniBand UD is required to support modern MPI
implementations. An extrapolation from the current results to future SSD-based
storage systems provides evidence that the current approach will remain
practical in the exascale generation. This transparent checkpointing approach
is evaluated using a framework of the DMTCP checkpointing package. Results are
shown for HPCG (linear algebra), NAMD (molecular dynamics), and the NAS NPB
benchmarks. In tests up to 32,752 MPI processes on 32,752 CPU cores,
checkpointing of a computation with a 38 TB memory footprint in 11 minutes is
demonstrated. Runtime overhead is reduced to less than 1%. The approach is also
evaluated across three widely used MPI implementations.
