  The explosive growth of various types of big data and advances in AI
technologies have catalyzed a new type of workloads called multi-modal DNNs.
Multi-modal DNNs are capable of interpreting and reasoning about information
from multiple modalities, making them more applicable to real-world AI
scenarios. In recent research, multi-modal DNNs have outperformed the best
uni-modal DNN in a wide range of distributed computing applications from
traditional multimedia systems to emerging autonomous edge systems. However,
despite their importance and superiority, very limited research attention has
been devoted to understand the characteristics of multi-modal DNNs and their
implications on current computing software/hardware platforms. Existing
benchmarks either target uni-modal DNNs or only focus on the algorithm
characteristics of multi-modal DNNs. There lacks representative benchmark
suites that provide comprehensive system and architecture level analysis of
multi-modal networks.
  To advance the understanding of these multi-modal DNN workloads and
facilitate related research, we present MMBench, an open-source, end-to-end
benchmark suite consisting of a set of real-world multi-modal DNN workloads
with relevant performance metrics for evaluation. We then use MMBench to
conduct an in-depth analysis on the characteristics of multi-modal DNNs. We
demonstrate their unique characteristics of clear multi-stage execution,
frequent synchronization and high heterogeneity, which distinguish them from
conventional uni-modal DNNs. Finally, we conduct a case study and extend our
benchmark to edge devices. We hope that our work can provide insights for
future software/hardware design and optimization to underpin multi-modal DNNs
on both cloud and edge computing platforms.
