  In recent years, transformer-based models have shown state-of-the-art results
for Natural Language Processing (NLP). In particular, the introduction of the
BERT language model brought with it breakthroughs in tasks such as question
answering and natural language inference, advancing applications that allow
humans to interact naturally with embedded devices. FPGA-based overlay
processors have been shown as effective solutions for edge image and video
processing applications, which mostly rely on low precision linear matrix
operations. In contrast, transformer-based NLP techniques employ a variety of
higher precision nonlinear operations with significantly higher frequency. We
present NPE, an FPGA-based overlay processor that can efficiently execute a
variety of NLP models. NPE offers software-like programmability to the end user
and, unlike FPGA designs that implement specialized accelerators for each
nonlinear function, can be upgraded for future NLP models without requiring
reconfiguration. We demonstrate that NPE can meet real-time conversational AI
latency targets for the BERT language model with $4\times$ lower power than
CPUs and $6\times$ lower power than GPUs. We also show NPE uses $3\times$ fewer
FPGA resources relative to comparable BERT network-specific accelerators in the
literature. NPE provides a cost-effective and power-efficient FPGA-based
solution for Natural Language Processing at the edge.
