  Compute-in-memory (CiM) has emerged as a compelling solution to alleviate
high data movement costs in von Neumann machines. CiM can perform massively
parallel general matrix multiplication (GEMM) operations in memory, the
dominant computation in Machine Learning (ML) inference. However, re-purposing
memory for compute poses key questions on 1) What type of CiM to use: Given a
multitude of analog and digital CiMs, determining their suitability from
systems perspective is needed. 2) When to use CiM: ML inference includes
workloads with a variety of memory and compute requirements, making it
difficult to identify when CiM is more beneficial than standard processing
cores. 3) Where to integrate CiM: Each memory level has different bandwidth and
capacity, that affects the data movement and locality benefits of CiM
integration.
  In this paper, we explore answers to these questions regarding CiM
integration for ML inference acceleration. We use Timeloop-Accelergy for early
system-level evaluation of CiM prototypes, including both analog and digital
primitives. We integrate CiM into different cache memory levels in an Nvidia
A100-like baseline architecture and tailor the dataflow for various ML
workloads. Our experiments show CiM architectures improve energy efficiency,
achieving up to 0.12x lower energy than the established baseline with INT-8
precision, and upto 4x performance gains with weight interleaving and
duplication. The proposed work provides insights into what type of CiM to use,
and when and where to optimally integrate it in the cache hierarchy for GEMM
acceleration.
