  Decoder-only Transformer models such as GPT have demonstrated superior
performance in text generation, by autoregressively predicting the next token.
However, the performance of GPT is bounded by low compute-to-memory-ratio and
high memory access. Throughput-oriented architectures such as GPUs target
parallel processing rather than sequential token generation, and are not
efficient for GPT acceleration, particularly on-device inference applications.
Process-in-memory (PIM) architectures can significantly reduce data movement
and provide high computation parallelism, and are promising candidates to
accelerate GPT inference.
  In this work, we propose PIM-GPT that aims to achieve high throughput, high
energy efficiency and end-to-end acceleration of GPT inference. PIM-GPT
leverages DRAM-based PIM solutions to perform multiply-accumulate (MAC)
operations on the DRAM chips, greatly reducing data movement. A compact
application-specific integrated chip (ASIC) is designed and synthesized to
initiate instructions to PIM chips and support data communication along with
necessary arithmetic computations. At the software level, the mapping scheme is
designed to maximize data locality and computation parallelism by partitioning
a matrix among DRAM channels and banks to utilize all in-bank computation
resources concurrently. We develop an event-driven clock-cycle accurate
simulator to validate the efficacy of the proposed PIM-GPT architecture.
Overall, PIM-GPT achieves 41$-$137$\times$, 631$-$1074$\times$ speedup and
339$-$1085$\times$, 890$-$1632$\times$ energy efficiency over GPU and CPU
baseline, respectively, on 8 GPT models with up to 1.4 billion parameters.
