Can you please train it on regular text?
Can you please train it on regular text for conversations
Do you mean you want an "instruct" version of the model?
Something like https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k or https://huggingface.co/datasets/Open-Orca/OpenOrca instead of stories? To see if it can generate coherent meaningful text? Or if you could share how to do it also would help.
To see if it can generate coherent meaningful text?
If I just train it on chat data, that won't magically make the model generate "coherent meaningful text." Right now the model is pretty undertrained, but with 41M parameters it's a miracle that it can generate even a slightly coherent sentence. If you still want to try, you can have a look at my Github repo and the steps should mostly be:
- Create a chat template. (You can use other models' chat templates too)
- Load whatever chat dataset you want to train on.
- Train it on the chat-templated texts.