IParraMartin's picture
End of training
1414952 verified
metadata
library_name: transformers
tags:
  - generated_from_trainer
model-index:
  - name: impossible-llms-english-random-trigram
    results: []

impossible-llms-english-random-trigram

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 4.3113

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 12
  • eval_batch_size: 8
  • seed: 0
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 384
  • total_eval_batch_size: 32
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • training_steps: 3000
  • mixed_precision_training: Native AMP
  • label_smoothing_factor: 0.1

Training results

Training Loss Epoch Step Validation Loss
14.1482 1.0 96 6.9646
11.4328 2.0 192 5.6992
11.1488 3.0 288 5.5112
10.5646 4.0 384 5.2485
10.2163 5.0 480 5.0376
9.8751 6.0 576 4.8854
9.6552 7.0 672 4.7683
9.4312 8.0 768 4.6836
9.301 9.0 864 4.6148
9.2448 10.0 960 4.5597
9.1271 11.0 1056 4.5156
9.0854 12.0 1152 4.4794
8.9255 13.0 1248 4.4493
8.8784 14.0 1344 4.4255
8.7833 15.0 1440 4.4035
8.6755 16.0 1536 4.3862
8.6895 17.0 1632 4.3722
8.6269 18.0 1728 4.3582
8.5067 19.0 1824 4.3492
8.4444 20.0 1920 4.3404
8.5608 21.0 2016 4.3332
8.4592 22.0 2112 4.3274
8.4261 23.0 2208 4.3233
8.471 24.0 2304 4.3193
8.3813 25.0 2400 4.3163
8.3404 26.0 2496 4.3149
8.3891 27.0 2592 4.3132
8.3628 28.0 2688 4.3122
8.4306 29.0 2784 4.3117
8.2589 30.0 2880 4.3113
8.247 31.0 2976 4.3113
33.3577 31.2520 3000 4.3113

Framework versions

  • Transformers 4.49.0
  • Pytorch 2.4.0+cu121
  • Datasets 3.4.0
  • Tokenizers 0.21.0