  Relational graph neural networks (RGNNs) are graph neural networks (GNNs)
with dedicated structures for modeling the different types of nodes and/or
edges in heterogeneous graphs. While RGNNs have been increasingly adopted in
many real-world applications due to their versatility and accuracy, they pose
performance and system design challenges due to their inherent computation
patterns, gap between the programming interface and kernel APIs, and heavy
programming efforts in optimizing kernels caused by their coupling with data
layout and heterogeneity. To systematically address these challenges, we
propose Pigeon, a novel two-level intermediate representation (IR) and its code
generator framework, that (a) represents the key properties of the RGNN models
to bridge the gap between the programming interface and kernel APIs, (b)
decouples model semantics, data layout, and operators-specific optimization
from each other to reduce programming efforts, (c) expresses and leverages
optimization opportunities in inter-operator transforms, data layout, and
operator-specific schedules. By building on one general matrix multiply (GEMM)
template and a node/edge traversal template, Pigeon achieves up to 7.8x
speed-up in inference and 5.6x speed-up in training compared with the
state-of-the-art public systems in select models, i.e., RGCN, RGAT, HGT, when
running heterogeneous graphs provided by Deep Graph Library (DGL) and Open
Graph Benchmark (OGB). Pigeon also triggers fewer out-of-memory (OOM) errors.
In addition, we propose linear operator fusion and compact materialization to
further accelerate the system by up to 2.2x.
