|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: |
|
|
- OuteAI/Lite-Oute-1-65M-Instruct |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# yujiepan/microllama-0.06B |
|
|
|
|
|
This is the same model as [OuteAI/Lite-Oute-1-65M-Instruct](https://huggingface.co/OuteAI/Lite-Oute-1-65M-Instruct/) but is converted in FP16. |
|
|
|
|
|
It is a small pretrained model that can do text generation. Very useful for algorithm development / debugging. |
|
|
|
|
|
Special thanks to the original author [OuteAI](https://huggingface.co/OuteAI/) for the hard work and contribution. |
|
|
|
|
|
**This repo is just a backup for myself. If you find this model useful, consider using the original repo instead.** |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
```bash |
|
|
lm_eval --model hf \ |
|
|
--model_args pretrained=yujiepan/microllama-0.06B,max_length=2048,dtype="<dtype>" \ |
|
|
--tasks wikitext \ |
|
|
--device cuda:0 \ |
|
|
--batch_size 1 |
|
|
``` |
|
|
|
|
|
| Model dtype | Word perplexity | |
|
|
| ----------- | --------------- | |
|
|
| FP32 | 59.1905 | |
|
|
| BF16 | 59.1187 | |
|
|
| FP16 | 59.1902 | |
|
|
|
|
|
Tested on A100 with `lm-eval==0.4.7`. |
|
|
|
