  Dense matrix multiply (MM) serves as one of the most heavily used kernels in
deep learning applications. To cope with the high computation demands of these
applications, heterogeneous architectures featuring both FPGA and dedicated
ASIC accelerators have emerged as promising platforms. For example, the
AMD/Xilinx Versal ACAP architecture combines general-purpose CPU cores and
programmable logic with AI Engine processors optimized for AI/ML. With 400
AIEs, it provides up to 6.4 TFLOPs performance for 32-bit floating-point data.
However, machine learning models often contain both large and small MM
operations. While large MM operations can be parallelized efficiently across
many cores, small MM operations typically cannot. We observe that executing
some small MM layers from the BERT natural language processing model on a
large, monolithic MM accelerator in Versal ACAP achieved less than 5% of the
theoretical peak performance. Therefore, one key question arises: How can we
design accelerators to fully use the abundant computation resources under
limited communication bandwidth for applications with multiple MM layers of
diverse sizes? We identify the biggest system throughput bottleneck resulting
from the mismatch of massive computation resources of one monolithic
accelerator and the various MM layers of small sizes in the application. To
resolve this problem, we propose the CHARM framework to compose multiple
diverse MM accelerator architectures working concurrently towards different
layers in one application. We deploy the CHARM framework for four different
applications, including BERT, ViT, NCF, MLP, on the AMD Versal ACAP VCK190
evaluation board. Our experiments show that we achieve 1.46 TFLOPs, 1.61
TFLOPs, 1.74 TFLOPs, and 2.94 TFLOPs inference throughput for BERT, ViT, NCF
and MLP, which obtain 5.40x, 32.51x, 1.00x and 1.00x throughput gains compared
to one monolithic accelerator.
