  Resistive Random-Access Memory (RRAM) is well-suited to accelerate neural
network (NN) workloads as RRAM-based Processing-in-Memory (PIM) architectures
natively support highly-parallel multiply-accumulate (MAC) operations that form
the backbone of most NN workloads. Unfortunately, NN workloads such as
transformers require support for non-MAC operations (e.g., softmax) that RRAM
cannot provide natively. Consequently, state-of-the-art works either integrate
additional digital logic circuits to support the non-MAC operations or offload
the non-MAC operations to CPU/GPU, resulting in significant performance and
energy efficiency overheads due to data movement.
  In this work, we propose NEON, a novel compiler optimization to enable the
end-to-end execution of the NN workload in RRAM. The key idea of NEON is to
transform each non-MAC operation into a lightweight yet highly-accurate neural
network. Utilizing neural networks to approximate the non-MAC operations
provides two advantages: 1) We can exploit the key strength of RRAM, i.e.,
highly-parallel MAC operation, to flexibly and efficiently execute non-MAC
operations in memory. 2) We can simplify RRAM's microarchitecture by
eliminating the additional digital logic circuits while reducing the data
movement overheads. Acceleration of the non-MAC operations in memory enables
NEON to achieve a 2.28x speedup compared to an idealized digital logic-based
RRAM. We analyze the trade-offs associated with the transformation and
demonstrate feasible use cases for NEON across different substrates.
