Qwen2.5-7B-Instruct - OpenThoughts3
This model is the final checkpoint from the exp2199b_redo3 experiment, which fine-tunes Qwen2.5-7B-Instruct on the OpenThoughts3 dataset.
Model Details
- Base Model: Qwen/Qwen2.5-7B-Instruct
- Training Dataset: OpenThoughts3-1.2M (1.2M examples)
- Final Checkpoint: step-11718
Training Hyperparameters
| Parameter | Value |
|---|---|
| Epochs | 5 |
| Batch Size | 512 |
| Learning Rate | 8e-5 |
| Max Sequence Length | 16384 |
| LR Schedule | Cosine |
| Warmup | 10% |
| Decay | 0.9 |
| Weight Decay | 0.0 |
| Beta1 | 0.9 |
| Beta2 | 0.999 |
| Hardware | TPU v4-512 |
Training Notes
- Era shuffling enabled (dataset shuffled every epoch)
- RoPE theta set to 1,000,000 for extended context support
- Fixed Qwen2.5 chat template applied
- Downloads last month
- 3
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support