  As deep learning is pervasive in modern applications, many deep learning
frameworks are presented for deep learning practitioners to develop and train
DNN models rapidly. Meanwhile, as training large deep learning models becomes a
trend in recent years, the training throughput and memory footprint are getting
crucial. Accordingly, optimizing training workloads with compiler optimizations
is inevitable and getting more and more attentions. However, existing deep
learning compilers (DLCs) mainly target inference and do not incorporate
holistic optimizations, such as automatic differentiation and automatic mixed
precision, in training workloads.
  In this paper, we present RAF, a deep learning compiler for training. Unlike
existing DLCs, RAF accepts a forward model and in-house generates a training
graph. Accordingly, RAF is able to systematically consolidate graph
optimizations for performance, memory and distributed training. In addition, to
catch up to the state-of-the-art performance with hand-crafted kernel libraries
as well as tensor compilers, RAF proposes an operator dialect mechanism to
seamlessly integrate all possible kernel implementations. We demonstrate that
by in-house training graph generation and operator dialect mechanism, we are
able to perform holistic optimizations and achieve either better training
throughput or larger batch size against PyTorch (eager and torchscript mode),
XLA, and DeepSpeed for popular transformer models on GPUs.
