  The Transformer has been an indispensable staple in deep learning. However,
for real-life applications, it is very challenging to deploy efficient
Transformers due to immense parameters and operations of models. To relieve
this burden, exploiting sparsity is an effective approach to accelerate
Transformers. Newly emerging Ampere GPUs leverage a 2:4 sparsity pattern to
achieve model acceleration, while it can hardly meet the diverse algorithm and
hardware constraints when deploying models. By contrast, we propose an
algorithm-hardware co-optimized framework to flexibly and efficiently
accelerate Transformers by utilizing general N:M sparsity patterns. (1) From
algorithm perspective, we propose a sparsity inheritance mechanism along with
an inherited dynamic pruning (IDP) method to obtain a series of N:M sparse
candidate Transformers rapidly. A model compression scheme is further proposed
to significantly reduce the storage requirement for deployment. (2) From
hardware perspective, we present a flexible and efficient hardware
architecture, namely STA, to achieve significant speedup when deploying N:M
sparse Transformers. STA features not only a computing engine unifying both
sparse-dense and dense-dense matrix multiplications with high computational
efficiency but also a scalable softmax module eliminating the latency from
intermediate off-chip data communication. Experimental results show that
compared to other methods, N:M sparse Transformers, generated using IDP,
achieves an average of 6.7% improvement on accuracy with high training
efficiency. Moreover, STA can achieve 14.47x and 11.33x speedup compared to
Intel i9-9900X and NVIDIA RTX 2080 Ti, respectively, and perform 2.00-19.47x
faster inference than the state-of-the-art FPGA-based accelerators for
Transformers.
