  Major data centre providers are introducing RDMA-based networks for their
tenants, as well as for operating the underlying infrastructure. In comparison
to traditional socket-based network stacks, RDMA-based networks offer higher
throughput, lower latency and reduced CPU overhead. However, transparent
checkpoint and migration operations become much more difficult. The key reason
is that the OS is removed from the critical path of communication. As a result,
some of the communication state itself resides in the NIC hardware and is no
more under the direct control of the OS. This control includes especially the
support for virtualisation of communication which is needed for live migration
of communication partners. In this paper, we propose the basic principles
required to implement a migration-capable RDMA-based network. We recommend some
changes at the software level and small changes at the hardware level. As a
proof of concept, we integrate the proposed changes into SoftRoCE, an
open-source kernel-level implementation of the RoCE protocol. We claim that
these changes introduce no runtime overhead when migration does not happen.
Finally, we develop a proof-of-concept implementation for migrating
containerised applications that use RDMA-based networks.
