ompress large pre-trained Transformer models</span>
<span class="sd">    by distilling the self-attention module of the last Transformer layer. The compressed model</span>
<span class="sd">    (called MiniLM) can be trained with task-agnostic distillation and then fine-tuned on various</span>
<span class="sd">    downstream tasks. We evaluate MiniLM on the GLUE benchmark and show that it achieves comparable</span>
<span class="sd">    results with BERT-base while being 4.3x smaller and 5.5x faster. We also 