running_log.txt

[INFO|2025-07-09 11:23:55] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/config.json

[INFO|2025-07-09 11:23:55] configuration_utils.py:768 >> Model config LlamaConfig {
  "_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "eos_token_id": [
    128001,
    128008,
    128009
  ],
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.2",
  "use_cache": true,
  "vocab_size": 128256
}


[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2034 >> loading file tokenizer.json from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/tokenizer.json

[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2034 >> loading file tokenizer.model from cache at None

[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2034 >> loading file added_tokens.json from cache at None

[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2034 >> loading file special_tokens_map.json from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/special_tokens_map.json

[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2034 >> loading file tokenizer_config.json from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/tokenizer_config.json

[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2034 >> loading file chat_template.jinja from cache at None

[INFO|2025-07-09 11:23:55] tokenization_utils_base.py:2304 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

[INFO|2025-07-09 11:23:55] logging.py:157 >> Add pad token: <|eot_id|>

[INFO|2025-07-09 11:23:55] logging.py:157 >> Add <|eot_id|>,<|eom_id|> to stop words.

[INFO|2025-07-09 11:23:55] logging.py:157 >> Loading dataset Codes3_query_filtered_553474_mark_less_than_8.0.json...

[INFO|2025-07-09 11:24:23] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/config.json

[INFO|2025-07-09 11:24:23] configuration_utils.py:768 >> Model config LlamaConfig {
  "_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "eos_token_id": [
    128001,
    128008,
    128009
  ],
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.2",
  "use_cache": true,
  "vocab_size": 128256
}


[WARNING|2025-07-09 11:24:23] logging.py:162 >> Input length is smaller than max length. Consider increase input length.

[INFO|2025-07-09 11:24:23] logging.py:157 >> Using llama3 scaling strategy and setting scaling factor to 1.0.

[INFO|2025-07-09 11:24:23] logging.py:157 >> Using block diagonal attention for sequence packing without cross-attention.

[INFO|2025-07-09 11:24:23] logging.py:157 >> Liger kernel has been applied to the model.

[INFO|2025-07-09 11:24:23] modeling_utils.py:3904 >> loading weights file model.safetensors from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/model.safetensors.index.json

[INFO|2025-07-09 11:24:23] modeling_utils.py:1582 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16.

[INFO|2025-07-09 11:24:23] configuration_utils.py:1140 >> Generate config GenerationConfig {
  "bos_token_id": 128000,
  "eos_token_id": [
    128001,
    128008,
    128009
  ]
}


[INFO|2025-07-09 11:24:33] modeling_utils.py:4888 >> All model checkpoint weights were used when initializing LlamaForCausalLM.


[INFO|2025-07-09 11:24:33] modeling_utils.py:4896 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at meta-llama/Llama-3.1-8B-Instruct.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training.

[INFO|2025-07-09 11:24:33] configuration_utils.py:1095 >> loading configuration file generation_config.json from cache at /home/kiho/.cache/huggingface/hub/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659/generation_config.json

[INFO|2025-07-09 11:24:33] configuration_utils.py:1140 >> Generate config GenerationConfig {
  "bos_token_id": 128000,
  "do_sample": true,
  "eos_token_id": [
    128001,
    128008,
    128009
  ],
  "temperature": 0.6,
  "top_p": 0.9
}


[INFO|2025-07-09 11:24:33] logging.py:157 >> Gradient checkpointing enabled.

[INFO|2025-07-09 11:24:33] logging.py:157 >> Using torch SDPA for faster training and inference.

[INFO|2025-07-09 11:24:33] logging.py:157 >> Upcasting trainable params to float32.

[INFO|2025-07-09 11:24:33] logging.py:157 >> Fine-tuning method: Freeze

[INFO|2025-07-09 11:24:33] logging.py:157 >> Set trainable layers: .15.,.31.

[INFO|2025-07-09 11:24:33] logging.py:157 >> trainable params: 436,224,000 || all params: 8,030,261,248 || trainable%: 5.4323

[INFO|2025-07-09 11:24:33] trainer.py:741 >> Using auto half precision backend

[INFO|2025-07-09 11:24:33] logging.py:157 >> Found linear modules: up_proj,v_proj,k_proj,down_proj,gate_proj,o_proj,q_proj

[INFO|2025-07-09 11:24:33] logging.py:157 >> Using APOLLO optimizer with args: {'rank': 256, 'proj': 'random', 'proj_type': 'std', 'update_proj_gap': 200, 'scale': 1, 'scale_type': 'channel', 'scale_front': False}.

[INFO|2025-07-09 11:24:34] trainer.py:2369 >> ***** Running training *****

[INFO|2025-07-09 11:24:34] trainer.py:2370 >>   Num examples = 21,829

[INFO|2025-07-09 11:24:34] trainer.py:2371 >>   Num Epochs = 1

[INFO|2025-07-09 11:24:34] trainer.py:2372 >>   Instantaneous batch size per device = 16

[INFO|2025-07-09 11:24:34] trainer.py:2375 >>   Total train batch size (w. parallel, distributed & accumulation) = 384

[INFO|2025-07-09 11:24:34] trainer.py:2376 >>   Gradient Accumulation steps = 8

[INFO|2025-07-09 11:24:34] trainer.py:2377 >>   Total optimization steps = 56

[INFO|2025-07-09 11:24:34] trainer.py:2378 >>   Number of trainable parameters = 436,224,000

[INFO|2025-07-09 11:27:27] logging.py:157 >> {'loss': 1.2229, 'learning_rate': 4.9961e-05, 'epoch': 0.02, 'throughput': 9132.51}

[INFO|2025-07-09 11:30:13] logging.py:157 >> {'loss': 0.9464, 'learning_rate': 4.9843e-05, 'epoch': 0.04, 'throughput': 9289.31}

[INFO|2025-07-09 11:33:00] logging.py:157 >> {'loss': 0.8654, 'learning_rate': 4.9647e-05, 'epoch': 0.05, 'throughput': 9339.94}

[INFO|2025-07-09 11:35:46] logging.py:157 >> {'loss': 0.7898, 'learning_rate': 4.9373e-05, 'epoch': 0.07, 'throughput': 9372.91}

[INFO|2025-07-09 11:38:33] logging.py:157 >> {'loss': 0.7780, 'learning_rate': 4.9023e-05, 'epoch': 0.09, 'throughput': 9387.49}

[INFO|2025-07-09 11:41:18] logging.py:157 >> {'loss': 0.7518, 'learning_rate': 4.8597e-05, 'epoch': 0.11, 'throughput': 9404.07}

[INFO|2025-07-09 11:44:04] logging.py:157 >> {'loss': 0.7607, 'learning_rate': 4.8097e-05, 'epoch': 0.12, 'throughput': 9417.27}

[INFO|2025-07-09 11:46:50] logging.py:157 >> {'loss': 0.7446, 'learning_rate': 4.7524e-05, 'epoch': 0.14, 'throughput': 9426.12}

[INFO|2025-07-09 11:49:36] logging.py:157 >> {'loss': 0.7510, 'learning_rate': 4.6881e-05, 'epoch': 0.16, 'throughput': 9430.98}

[INFO|2025-07-09 11:52:23] logging.py:157 >> {'loss': 0.7473, 'learning_rate': 4.6168e-05, 'epoch': 0.18, 'throughput': 9430.19}

[INFO|2025-07-09 11:55:09] logging.py:157 >> {'loss': 0.7245, 'learning_rate': 4.5389e-05, 'epoch': 0.19, 'throughput': 9431.67}

[INFO|2025-07-09 11:57:56] logging.py:157 >> {'loss': 0.7104, 'learning_rate': 4.4546e-05, 'epoch': 0.21, 'throughput': 9433.81}

[INFO|2025-07-09 12:00:42] logging.py:157 >> {'loss': 0.7083, 'learning_rate': 4.3641e-05, 'epoch': 0.23, 'throughput': 9434.95}

[INFO|2025-07-09 12:03:28] logging.py:157 >> {'loss': 0.7093, 'learning_rate': 4.2678e-05, 'epoch': 0.25, 'throughput': 9436.03}

[INFO|2025-07-09 12:06:15] logging.py:157 >> {'loss': 0.7082, 'learning_rate': 4.1659e-05, 'epoch': 0.26, 'throughput': 9436.39}

[INFO|2025-07-09 12:09:01] logging.py:157 >> {'loss': 0.7060, 'learning_rate': 4.0587e-05, 'epoch': 0.28, 'throughput': 9437.10}

[INFO|2025-07-09 12:11:48] logging.py:157 >> {'loss': 0.7018, 'learning_rate': 3.9467e-05, 'epoch': 0.30, 'throughput': 9437.39}

[INFO|2025-07-09 12:14:35] logging.py:157 >> {'loss': 0.7065, 'learning_rate': 3.8301e-05, 'epoch': 0.32, 'throughput': 9437.86}

[INFO|2025-07-09 12:17:21] logging.py:157 >> {'loss': 0.7104, 'learning_rate': 3.7093e-05, 'epoch': 0.33, 'throughput': 9438.94}

[INFO|2025-07-09 12:20:07] logging.py:157 >> {'loss': 0.6914, 'learning_rate': 3.5847e-05, 'epoch': 0.35, 'throughput': 9440.93}

[INFO|2025-07-09 12:22:53] logging.py:157 >> {'loss': 0.6903, 'learning_rate': 3.4567e-05, 'epoch': 0.37, 'throughput': 9442.34}

[INFO|2025-07-09 12:25:39] logging.py:157 >> {'loss': 0.6827, 'learning_rate': 3.3257e-05, 'epoch': 0.39, 'throughput': 9444.06}

[INFO|2025-07-09 12:28:25] logging.py:157 >> {'loss': 0.6886, 'learning_rate': 3.1921e-05, 'epoch': 0.40, 'throughput': 9445.17}

[INFO|2025-07-09 12:31:11] logging.py:157 >> {'loss': 0.6824, 'learning_rate': 3.0563e-05, 'epoch': 0.42, 'throughput': 9447.06}

[INFO|2025-07-09 12:33:57] logging.py:157 >> {'loss': 0.6974, 'learning_rate': 2.9188e-05, 'epoch': 0.44, 'throughput': 9447.34}

[INFO|2025-07-09 12:36:43] logging.py:157 >> {'loss': 0.6863, 'learning_rate': 2.7799e-05, 'epoch': 0.46, 'throughput': 9447.47}

[INFO|2025-07-09 12:39:29] logging.py:157 >> {'loss': 0.6833, 'learning_rate': 2.6402e-05, 'epoch': 0.47, 'throughput': 9448.79}

[INFO|2025-07-09 12:42:15] logging.py:157 >> {'loss': 0.6754, 'learning_rate': 2.5000e-05, 'epoch': 0.49, 'throughput': 9449.45}

[INFO|2025-07-09 12:45:02] logging.py:157 >> {'loss': 0.6524, 'learning_rate': 2.3598e-05, 'epoch': 0.51, 'throughput': 9448.98}

[INFO|2025-07-09 12:47:48] logging.py:157 >> {'loss': 0.6632, 'learning_rate': 2.2201e-05, 'epoch': 0.53, 'throughput': 9449.37}

[INFO|2025-07-09 12:50:35] logging.py:157 >> {'loss': 0.6564, 'learning_rate': 2.0812e-05, 'epoch': 0.55, 'throughput': 9449.77}

[INFO|2025-07-09 12:53:21] logging.py:157 >> {'loss': 0.6817, 'learning_rate': 1.9437e-05, 'epoch': 0.56, 'throughput': 9449.98}

[INFO|2025-07-09 12:56:07] logging.py:157 >> {'loss': 0.6662, 'learning_rate': 1.8079e-05, 'epoch': 0.58, 'throughput': 9451.06}

[INFO|2025-07-09 12:58:53] logging.py:157 >> {'loss': 0.7141, 'learning_rate': 1.6743e-05, 'epoch': 0.60, 'throughput': 9451.69}

[INFO|2025-07-09 13:01:39] logging.py:157 >> {'loss': 0.6738, 'learning_rate': 1.5433e-05, 'epoch': 0.62, 'throughput': 9451.95}

[INFO|2025-07-09 13:04:25] logging.py:157 >> {'loss': 0.6843, 'learning_rate': 1.4153e-05, 'epoch': 0.63, 'throughput': 9452.04}

[INFO|2025-07-09 13:07:11] logging.py:157 >> {'loss': 0.6534, 'learning_rate': 1.2907e-05, 'epoch': 0.65, 'throughput': 9452.47}

[INFO|2025-07-09 13:09:58] logging.py:157 >> {'loss': 0.6720, 'learning_rate': 1.1699e-05, 'epoch': 0.67, 'throughput': 9452.73}

[INFO|2025-07-09 13:12:44] logging.py:157 >> {'loss': 0.6588, 'learning_rate': 1.0533e-05, 'epoch': 0.69, 'throughput': 9453.31}

[INFO|2025-07-09 13:15:30] logging.py:157 >> {'loss': 0.6472, 'learning_rate': 9.4128e-06, 'epoch': 0.70, 'throughput': 9453.20}

[INFO|2025-07-09 13:18:17] logging.py:157 >> {'loss': 0.6778, 'learning_rate': 8.3413e-06, 'epoch': 0.72, 'throughput': 9453.07}

[INFO|2025-07-09 13:21:03] logging.py:157 >> {'loss': 0.6481, 'learning_rate': 7.3223e-06, 'epoch': 0.74, 'throughput': 9452.75}

[INFO|2025-07-09 13:23:50] logging.py:157 >> {'loss': 0.6583, 'learning_rate': 6.3589e-06, 'epoch': 0.76, 'throughput': 9452.46}

[INFO|2025-07-09 13:26:37] logging.py:157 >> {'loss': 0.6456, 'learning_rate': 5.4542e-06, 'epoch': 0.77, 'throughput': 9451.75}

[INFO|2025-07-09 13:29:24] logging.py:157 >> {'loss': 0.6864, 'learning_rate': 4.6110e-06, 'epoch': 0.79, 'throughput': 9450.93}

[INFO|2025-07-09 13:32:11] logging.py:157 >> {'loss': 0.6776, 'learning_rate': 3.8319e-06, 'epoch': 0.81, 'throughput': 9450.28}

[INFO|2025-07-09 13:34:58] logging.py:157 >> {'loss': 0.6744, 'learning_rate': 3.1194e-06, 'epoch': 0.83, 'throughput': 9449.46}

[INFO|2025-07-09 13:37:44] logging.py:157 >> {'loss': 0.6586, 'learning_rate': 2.4758e-06, 'epoch': 0.84, 'throughput': 9449.35}

[INFO|2025-07-09 13:40:32] logging.py:157 >> {'loss': 0.6771, 'learning_rate': 1.9030e-06, 'epoch': 0.86, 'throughput': 9448.50}

[INFO|2025-07-09 13:43:19] logging.py:157 >> {'loss': 0.6512, 'learning_rate': 1.4029e-06, 'epoch': 0.88, 'throughput': 9447.88}

[INFO|2025-07-09 13:46:06] logging.py:157 >> {'loss': 0.6667, 'learning_rate': 9.7707e-07, 'epoch': 0.90, 'throughput': 9447.43}

[INFO|2025-07-09 13:48:53] logging.py:157 >> {'loss': 0.6546, 'learning_rate': 6.2680e-07, 'epoch': 0.91, 'throughput': 9446.95}

[INFO|2025-07-09 13:51:39] logging.py:157 >> {'loss': 0.6562, 'learning_rate': 3.5322e-07, 'epoch': 0.93, 'throughput': 9446.63}

[INFO|2025-07-09 13:54:26] logging.py:157 >> {'loss': 0.6517, 'learning_rate': 1.5719e-07, 'epoch': 0.95, 'throughput': 9446.24}

[INFO|2025-07-09 13:57:13] logging.py:157 >> {'loss': 0.6336, 'learning_rate': 3.9330e-08, 'epoch': 0.97, 'throughput': 9446.33}

[INFO|2025-07-09 14:00:00] logging.py:157 >> {'loss': 0.6480, 'learning_rate': 0.0000e+00, 'epoch': 0.98, 'throughput': 9445.79}

[INFO|2025-07-09 14:00:00] trainer.py:3910 >> Saving model checkpoint to saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/checkpoint-56

[INFO|2025-07-09 14:00:00] configuration_utils.py:420 >> Configuration saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/checkpoint-56/config.json

[INFO|2025-07-09 14:00:00] configuration_utils.py:909 >> Configuration saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/checkpoint-56/generation_config.json

[INFO|2025-07-09 14:00:24] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/checkpoint-56/model.safetensors.index.json.

[INFO|2025-07-09 14:00:24] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/checkpoint-56/tokenizer_config.json

[INFO|2025-07-09 14:00:24] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/checkpoint-56/special_tokens_map.json

[INFO|2025-07-09 14:00:25] trainer.py:2643 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)



[INFO|2025-07-09 14:00:25] trainer.py:3910 >> Saving model checkpoint to saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx

[INFO|2025-07-09 14:00:25] configuration_utils.py:420 >> Configuration saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/config.json

[INFO|2025-07-09 14:00:25] configuration_utils.py:909 >> Configuration saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/generation_config.json

[INFO|2025-07-09 14:00:51] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/model.safetensors.index.json.

[INFO|2025-07-09 14:00:51] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/tokenizer_config.json

[INFO|2025-07-09 14:00:51] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/Llama-3.1-8B-Instruct/freeze/llama_under8_nlx/special_tokens_map.json

[WARNING|2025-07-09 14:00:51] logging.py:162 >> No metric eval_loss to plot.

[WARNING|2025-07-09 14:00:51] logging.py:162 >> No metric eval_accuracy to plot.

[INFO|2025-07-09 14:00:51] modelcard.py:449 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}