  Memristor crossbars are circuits capable of performing analog matrix-vector
multiplications, overcoming the fundamental energy efficiency limitations of
digital logic. They have been shown to be effective in special-purpose
accelerators for a limited set of neural network applications.
  We present the Programmable Ultra-efficient Memristor-based Accelerator
(PUMA) which enhances memristor crossbars with general purpose execution units
to enable the acceleration of a wide variety of Machine Learning (ML) inference
workloads. PUMA's microarchitecture techniques exposed through a specialized
Instruction Set Architecture (ISA) retain the efficiency of in-memory computing
and analog circuitry, without compromising programmability.
  We also present the PUMA compiler which translates high-level code to PUMA
ISA. The compiler partitions the computational graph and optimizes instruction
scheduling and register allocation to generate code for large and complex
workloads to run on thousands of spatial cores.
  We have developed a detailed architecture simulator that incorporates the
functionality, timing, and power models of PUMA's components to evaluate
performance and energy consumption. A PUMA accelerator running at 1 GHz can
reach area and power efficiency of $577~GOPS/s/mm^2$ and $837~GOPS/s/W$,
respectively. Our evaluation of diverse ML applications from image recognition,
machine translation, and language modelling (5M-800M synapses) shows that PUMA
achieves up to $2,446\times$ energy and $66\times$ latency improvement for
inference compared to state-of-the-art GPUs. Compared to an
application-specific memristor-based accelerator, PUMA incurs small energy
overheads at similar inference latency and added programmability.
