  Softmax is popular normalization method used in machine learning. Deep
learning solutions like Transformer or BERT use the softmax function
intensively, so it is worthwhile to optimize its performance. This article
presents our methodology of optimization and its results applied to softmax. By
presenting this methodology, we hope to increase an interest in deep learning
optimizations for CPUs. We believe that the optimization process presented here
could be transferred to other deep learning frameworks such as TensorFlow or
PyTorch.
