  We devise a performance model for GPU training of Deep Learning
Recommendation Models (DLRM), whose GPU utilization is low compared to other
well-optimized CV and NLP models. We show that both the device active time (the
sum of kernel runtimes) but also the device idle time are important components
of the overall device time. We therefore tackle them separately by (1) flexibly
adopting heuristic-based and ML-based kernel performance models for operators
that dominate the device active time, and (2) categorizing operator overheads
into five types to determine quantitatively their contribution to the device
active time. Combining these two parts, we propose a critical-path-based
algorithm to predict the per-batch training time of DLRM by traversing its
execution graph. We achieve less than 10% geometric mean average error (GMAE)
in all kernel performance modeling, and 4.61% and 7.96% geomean errors for GPU
active time and overall E2E per-batch training time prediction with overheads
from individual workloads, respectively. A slight increase of 2.19% incurred in
E2E prediction error with shared overheads across workloads suggests the
feasibility of using shared overheads in large-scale prediction. We show that
our general performance model not only achieves low prediction error on DLRM,
which has highly customized configurations and is dominated by multiple factors
but also yields comparable accuracy on other compute-bound ML models targeted
by most previous methods. Using this performance model and graph-level data and
task dependency analysis, we show our system can provide more general
model-system co-design than previous methods.
