  Despite the success of Transformers in self- supervised learning with
applications to various downstream tasks, the computational cost of training
and inference remains a major challenge for applying these models to a wide
spectrum of devices. Several isolated attempts have been made to compress
Transformers, but the settings and metrics are different across studies.
Trade-off at various compression rates are also largely missing in prior work,
making it difficult to compare compression techniques. In this work, we aim to
provide context for the isolated results, studying several commonly used
compression techniques, including weight pruning, head pruning, low-rank
approximation, and knowledge distillation. We report trade- off at various
compression rate, including wall-clock time, the number of parameters, and the
number of multiply-accumulate operations. Our results show that compared to
recent approaches, basic compression techniques are strong baselines. We
further present several applications of our results, revealing properties of
Transformers, such as the significance of diagonal attention heads. In
addition, our results lead to a simple combination of compression techniques
that improves trade-off over recent approaches. We hope the results would
promote more diverse comparisons among model compression techniques and promote
the use of model compression as a tool for analyzing models. Our code of
compressing speech self-supervised model is available at
https://github.com/nervjack2/Speech-SSL-Compression/.
