-Wikipedia-dump-for-pretraining"> </a></h3><p>ULMFiT can be peretrained on relativly small datasets - 100 million tokens are sufficient to get state-of-the art classification results (compared to Transformer models as BERT, which need huge amounts of training data). The easiest way is to pretrain a language model on Wikipedia.</p>
<p>The code for the preperation steps is heavily inspired by / copied from the <strong>fast.ai NLP-course</strong>: <a href="https://github.com/fastai/course-nlp/blob/master/nlput