ss="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="2.-Language-model-pretraining-on-Wikipedia-Dump">2. Language model pretraining on Wikipedia Dump<a class="anchor-link" href="#2.-Language-model-pretraining-on-Wikipedia-Dump"> </a></h3><p>Notebook: <code>2_ulmfit_lm_pretraining.ipynb</code></p>
<p>To get the best result, you can train two seperate language models - a forward and a backward model. You'll have to run the complete notebook twice and set the <code>backwards<