llada-1.0-s1

This model is a fine-tuned version of GSAI-ML/LLaDA-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3445

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 16
• total_eval_batch_size: 64
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
0.1327	1.6129	100	0.3969
0.5289	3.2258	200	0.3672
0.1107	4.8387	300	0.3297
1.3167	6.4516	400	0.3453
0.1186	8.0645	500	0.3398
0.8868	9.6774	600	0.3414
0.894	11.2903	700	0.3492
0.68	12.9032	800	0.3680
1.0221	14.5161	900	0.3508
0.191	16.1290	1000	0.3516
0.9078	17.7419	1100	0.3305
0.1158	19.3548	1200	0.3445

Framework versions

• PEFT 0.15.1
• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.3.2
• Tokenizers 0.21.4