  The recent years witness a trend of applying large-scale distributed deep
learning in both business and scientific computing areas, whose goal is to
speed up the training time to achieve a state-of-the-art quality. The HPC
community feels a great interest in building the HPC AI systems that are
dedicated to running those workloads. The HPC AI benchmarks accelerate the
process. Unfortunately, benchmarking HPC AI systems at scale raises serious
challenges. None of previous HPC AI benchmarks achieve the goal of being
equivalent, relevant, representative, affordable, and repeatable. This paper
presents a comprehensive methodology, tools, Roofline performance models, and
innovative metrics for benchmarking, optimizing, and ranking HPC AI systems,
which we call HPC AI500 V2.0. We abstract the HPC AI system into nine
independent layers, and present explicit benchmarking rules and procedures to
assure equivalence of each layer, repeatability, and replicability. On the
basis of AIBench -- by far the most comprehensive AI benchmarks suite, we
present and build two HPC AI benchmarks from both business and scientific
computing: Image Classification, and Extreme Weather Analytics, achieving both
representativeness and affordability. To rank the performance and
energy-efficiency of HPC AI systems, we propose Valid FLOPS, and Valid FLOPS
per watt, which impose a penalty on failing to achieve the target quality. We
propose using convolution and GEMM -- the two most intensively-used kernel
functions to measure the upper bound performance of the HPC AI systems, and
present HPC AI roofline models for guiding performance optimizations. The
evaluations show our methodology, benchmarks, performance models, and metrics
can measure, optimize, and rank the HPC AI systems in a scalable, simple, and
affordable way. HPC AI500 V2.0 are publicly available from
http://www.benchcouncil.org/benchhub/hpc-ai500-benchmark.
