  Transparently checkpointing MPI for fault tolerance and load balancing is a
long-standing problem in HPC. The problem has been complicated by the need to
provide checkpoint-restart services for all combinations of an MPI
implementation over all network interconnects. This work presents MANA
(MPI-Agnostic Network-Agnostic transparent checkpointing), a single code base
which supports all MPI implementation and interconnect combinations. The
agnostic properties imply that one can checkpoint an MPI application under one
MPI implementation and perhaps over TCP, and then restart under a second MPI
implementation over InfiniBand on a cluster with a different number of CPU
cores per node. This technique is based on a novel "split-process" approach,
which enables two separate programs to co-exist within a single process with a
single address space. This work overcomes the limitations of the two most
widely adopted transparent checkpointing solutions, BLCR and DMTCP/InfiniBand,
which require separate modifications to each MPI implementation and/or
underlying network API. The runtime overhead is found to be insignificant both
for checkpoint-restart within a single host, and when comparing a local MPI
computation that was migrated to a remote cluster against an ordinary MPI
computation running natively on that same remote cluster.
