  The Transformer architecture revolutionized the field of natural language
processing (NLP). Transformers-based models (e.g., BERT) power many important
Web services, such as search, translation, question-answering, etc. While
enormous research attention is paid to the training of those models, relatively
little efforts are made to improve their inference performance. This paper
comes to address this gap by presenting an empirical analysis of scalability
and performance of inferencing a Transformer-based model on CPUs. Focusing on
the highly popular BERT model, we identify key components of the Transformer
architecture where the bulk of the computation happens, and propose three
optimizations to speed them up. The optimizations are evaluated using the
inference benchmark from HuggingFace, and are shown to achieve the speedup of
up to x2.37. The considered optimizations do not require any changes to the
implementation of the models nor affect their accuracy.
