---
library_name: transformers
license: other
license_name: lfm1.0
license_link: LICENSE
language:
- en
- ar
- zh
- fr
- de
- ja
- ko
- es
pipeline_tag: text-generation
datasets:
- winglian/pirate-ultrachat-10k
base_model:
- LiquidAI/LFM2-1.2B
tags:
- lfm2
- liquid
- axolotl
- edge
---

# LFM2-1.2B-Pirate

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)

Fine-tuned model trained with [Axolotl](https://github.com/axolotl-ai-cloud/axolotl).

The process was very fast and efficient! It shows the fine-tuneability of [LiquidAI/LFM-1.2B](https://huggingface.co/LiquidAI/LFM2-1.2B).

Thanks to @winglian for writing the entire code and making LFM2 compatible with Axolotl! ❤️

**Example**:
```
<|startoftext|><|im_start|>user
What is C. elegans?<|im_end|>
<|im_start|>assistant
Ahoy, matey!  You be talkin' 'bout the little scurvy dog, eh?  Well, shiver me timbers, that's a fine question! 

C. elegans, they say, be a tiny, scurvy dog, about the size of a thumbnail, but they're as smart as a kraken's tentacles and as clever as a pirate's treasure map. They be a little worm, but they've got a brain as big as a ship's mast and they can navigate through treacherous waters with ease. 

They be used in all sorts of experiments, like a pirate's treasure hunt, to learn more about how life works.  They be like a puzzle, a puzzle, and a treasure chest all rolled into one!  So, if you want to know more about these little creatures, just keep your eyes peeled, and maybe even get a bit closer, matey! 
<|im_end|>
```

## 🧩 Configuration

```yaml
base_model: LiquidAI/LFM2-1.2B
xformers_attention: true
sample_packing: true
chunked_cross_entropy: true
learning_rate: 1e-5
sequence_len: 16384
micro_batch_size: 1
gradient_accumulation_steps: 2
gradient_checkpointing: true
optimizer: adamw_torch_8bit
lr_scheduler: cosine
warmup_ratio: 0.2
float16: true
bf16: false
max_grad_norm: 0.1
num_epochs: 3
saves_per_epoch: 1
logging_steps: 5
output_dir: ./outputs/lfm2-sft-pirate-rrr
chat_template: tokenizer_default
datasets:
  - path: winglian/pirate-ultrachat-10k
    type: chat_template
    split: train
eot_tokens:
  - "<|im_end|>"
dataloader_prefetch_factor: 8
dataloader_num_workers: 2
dataloader_pin_memory: true
```