  Progress in artificial intelligence and machine learning over the past decade
has been driven by the ability to train larger deep neural networks (DNNs),
leading to a compute demand that far exceeds the growth in hardware performance
afforded by Moore's law. Training DNNs is an extremely memory-intensive
process, requiring not just the model weights but also activations and
gradients for an entire minibatch to be stored. The need to provide
high-density and low-leakage on-chip memory motivates the exploration of
emerging non-volatile memory for training accelerators. Spin-Transfer-Torque
MRAM (STT-MRAM) offers several desirable properties for training accelerators,
including 3-4x higher density than SRAM, significantly reduced leakage power,
high endurance and reasonable access time. On the one hand, MRAM write
operations require high write energy and latency due to the need to ensure
reliable switching.
  In this study, we perform a comprehensive device-to-system evaluation and
co-optimization of STT-MRAM for efficient ML training accelerator design. We
devised a cross-layer simulation framework to evaluate the effectiveness of
STT-MRAM as a scratchpad replacing SRAM in a systolic-array-based DNN
accelerator. To address the inefficiency of writes in STT-MRAM, we propose to
reduce write voltage and duration. To evaluate the ensuing accuracy-efficiency
trade-off, we conduct a thorough analysis of the error tolerance of input
activations, weights, and errors during the training. We propose heterogeneous
memory configurations that enable training convergence with good accuracy. We
show that MRAM provide up to 15-22x improvement in system level energy across a
suite of DNN benchmarks under iso-capacity and iso-area scenarios. Further
optimizing STT-MRAM write operations can provide over 2x improvement in write
energy for minimal degradation in application-level training accuracy.
