, when distilling a BERT-base model to a 3-layer BERT, you could initialize the student model with <a class="reference external" href="#https://github.com/ymcui/Chinese-BERT-wwm">RBT3</a> (for Chinese tasks) or the first three layers of BERT (for English tasks) to avoid cold start problem.
We recommend that users use pre-trained student models whenever possible to fully take the advantage of large-scale pre-training.</p>
<p><strong>Q</strong>: How to set training hyperparamters for the distillation experime