eos_token
#5
by maymo - opened
In the tokenizer_config.json, the eos_token is set to "<|endoftext|>" and this token is not present in the chat_template.
Is "<|im_end|>" is the eos_token or "<|endoftext|>"?
I get this warning when fine tuning this model in Axolotl:
EOS token '<|endoftext|>' not found in chat_template. Please check if your template/EOS token is correct.