pecific task.</p>
<p>| Model                 | #Layers | Hidden_size | Feed-forward size | #Params | Relative size |
| :——————— | ——— | ———– | —————– | ——– | ————- |
| BERT-base-cased (teacher)  | 12        | 768         | 3072              | 108M     | 100%          |
| RoBERTa-wwm-ext (teacher) | 12        | 768         | 3072              | 108M     | 100%          |
| T6 (student)              | 6         | 768         | 3072              | 65M      