  Recent advancements in large language models (LLMs) boasting billions of
parameters have generated a significant demand for efficient deployment in
inference workloads. The majority of existing approaches rely on temporal
architectures that reuse hardware units for different network layers and
operators. However, these methods often encounter challenges in achieving low
latency due to considerable memory access overhead. This paper investigates the
feasibility and potential of model-specific spatial acceleration for LLM
inference on FPGAs. Our approach involves the specialization of distinct
hardware units for specific operators or layers, facilitating direct
communication between them through a dataflow architecture while minimizing
off-chip memory accesses. We introduce a comprehensive analytical model for
estimating the performance of a spatial LLM accelerator, taking into account
the on-chip compute and memory resources available on an FPGA. Through our
analysis, we can determine the scenarios in which FPGA-based spatial
acceleration can outperform its GPU-based counterpart. To enable more
productive implementations of an LLM model on FPGAs, we further provide a
library of high-level synthesis (HLS) kernels that are composable and reusable.
This library will be made available as open-source. To validate the
effectiveness of both our analytical model and HLS library, we have implemented
BERT and GPT2 on an AMD Alveo U280 FPGA device. Experimental results
demonstrate our approach can achieve up to 16.1x speedup when compared to
previous FPGA-based accelerators for the BERT model. For GPT generative
inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the
prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy
efficiency compared to the NVIDIA A100 GPU in the decode stage.
