Upload 12 files

.gitattributes

@@ -5,3 +5,4 @@ model-00004-of-00006.safetensors filter=lfs diff=lfs merge=lfs -text
 model-00002-of-00006.safetensors filter=lfs diff=lfs merge=lfs -text
 model-00003-of-00006.safetensors filter=lfs diff=lfs merge=lfs -text
 adapter_model.safetensors filter=lfs diff=lfs merge=lfs -text
+optimizer.pt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,204 @@
+---
+library_name: peft
+base_model: TinyPixel/CodeLlama-7B-Python-bf16-sharded
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+### Framework versions
+
+- PEFT 0.7.1

adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "TinyPixel/CodeLlama-7B-Python-bf16-sharded",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6aab4f18b1656b61fa36a1e8edc40ffab669f5b4516775c2c964b982be93990d
+size 33571624

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1b863c26745543031e6823721935ff961133b4a161cd899125c810f315e1595
+size 67217018

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": null,
+  "model_max_length": 300,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}

trainer_state.json

@@ -0,0 +1,1461 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 29.88888888888889,
+  "eval_steps": 500,
+  "global_step": 240,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.11,
+      "learning_rate": 7.4074074074074075e-06,
+      "loss": 1.0694,
+      "step": 1
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 1.4814814814814815e-05,
+      "loss": 1.0895,
+      "step": 2
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 2.2222222222222223e-05,
+      "loss": 1.112,
+      "step": 3
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 2.962962962962963e-05,
+      "loss": 1.1232,
+      "step": 4
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 3.7037037037037037e-05,
+      "loss": 1.1558,
+      "step": 5
+    },
+    {
+      "epoch": 0.67,
+      "learning_rate": 4.4444444444444447e-05,
+      "loss": 1.1461,
+      "step": 6
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 5.185185185185185e-05,
+      "loss": 1.0642,
+      "step": 7
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 5.925925925925926e-05,
+      "loss": 1.0218,
+      "step": 8
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 6.666666666666667e-05,
+      "loss": 1.1029,
+      "step": 9
+    },
+    {
+      "epoch": 1.22,
+      "learning_rate": 7.407407407407407e-05,
+      "loss": 1.0927,
+      "step": 10
+    },
+    {
+      "epoch": 1.33,
+      "learning_rate": 8.148148148148148e-05,
+      "loss": 1.0817,
+      "step": 11
+    },
+    {
+      "epoch": 1.44,
+      "learning_rate": 8.888888888888889e-05,
+      "loss": 1.0247,
+      "step": 12
+    },
+    {
+      "epoch": 1.56,
+      "learning_rate": 9.62962962962963e-05,
+      "loss": 0.9867,
+      "step": 13
+    },
+    {
+      "epoch": 1.67,
+      "learning_rate": 0.0001037037037037037,
+      "loss": 1.0072,
+      "step": 14
+    },
+    {
+      "epoch": 1.78,
+      "learning_rate": 0.00011111111111111112,
+      "loss": 1.0207,
+      "step": 15
+    },
+    {
+      "epoch": 1.89,
+      "learning_rate": 0.00011851851851851852,
+      "loss": 0.9365,
+      "step": 16
+    },
+    {
+      "epoch": 2.11,
+      "learning_rate": 0.00012592592592592592,
+      "loss": 0.9648,
+      "step": 17
+    },
+    {
+      "epoch": 2.22,
+      "learning_rate": 0.00013333333333333334,
+      "loss": 0.8807,
+      "step": 18
+    },
+    {
+      "epoch": 2.33,
+      "learning_rate": 0.00014074074074074076,
+      "loss": 0.8205,
+      "step": 19
+    },
+    {
+      "epoch": 2.44,
+      "learning_rate": 0.00014814814814814815,
+      "loss": 0.8341,
+      "step": 20
+    },
+    {
+      "epoch": 2.56,
+      "learning_rate": 0.00015555555555555556,
+      "loss": 0.8371,
+      "step": 21
+    },
+    {
+      "epoch": 2.67,
+      "learning_rate": 0.00016296296296296295,
+      "loss": 0.6997,
+      "step": 22
+    },
+    {
+      "epoch": 2.78,
+      "learning_rate": 0.00017037037037037037,
+      "loss": 0.7197,
+      "step": 23
+    },
+    {
+      "epoch": 2.89,
+      "learning_rate": 0.00017777777777777779,
+      "loss": 0.6475,
+      "step": 24
+    },
+    {
+      "epoch": 3.11,
+      "learning_rate": 0.0001851851851851852,
+      "loss": 0.5663,
+      "step": 25
+    },
+    {
+      "epoch": 3.22,
+      "learning_rate": 0.0001925925925925926,
+      "loss": 0.6429,
+      "step": 26
+    },
+    {
+      "epoch": 3.33,
+      "learning_rate": 0.0002,
+      "loss": 0.7241,
+      "step": 27
+    },
+    {
+      "epoch": 3.44,
+      "learning_rate": 0.0001991769547325103,
+      "loss": 0.5817,
+      "step": 28
+    },
+    {
+      "epoch": 3.56,
+      "learning_rate": 0.0001983539094650206,
+      "loss": 0.6,
+      "step": 29
+    },
+    {
+      "epoch": 3.67,
+      "learning_rate": 0.00019753086419753085,
+      "loss": 0.5576,
+      "step": 30
+    },
+    {
+      "epoch": 3.78,
+      "learning_rate": 0.00019670781893004114,
+      "loss": 0.4875,
+      "step": 31
+    },
+    {
+      "epoch": 3.89,
+      "learning_rate": 0.00019588477366255146,
+      "loss": 0.4722,
+      "step": 32
+    },
+    {
+      "epoch": 4.11,
+      "learning_rate": 0.00019506172839506175,
+      "loss": 0.5002,
+      "step": 33
+    },
+    {
+      "epoch": 4.22,
+      "learning_rate": 0.00019423868312757204,
+      "loss": 0.4247,
+      "step": 34
+    },
+    {
+      "epoch": 4.33,
+      "learning_rate": 0.00019341563786008233,
+      "loss": 0.4939,
+      "step": 35
+    },
+    {
+      "epoch": 4.44,
+      "learning_rate": 0.0001925925925925926,
+      "loss": 0.4732,
+      "step": 36
+    },
+    {
+      "epoch": 4.56,
+      "learning_rate": 0.00019176954732510288,
+      "loss": 0.4211,
+      "step": 37
+    },
+    {
+      "epoch": 4.67,
+      "learning_rate": 0.00019094650205761317,
+      "loss": 0.3932,
+      "step": 38
+    },
+    {
+      "epoch": 4.78,
+      "learning_rate": 0.00019012345679012346,
+      "loss": 0.3905,
+      "step": 39
+    },
+    {
+      "epoch": 4.89,
+      "learning_rate": 0.00018930041152263375,
+      "loss": 0.3095,
+      "step": 40
+    },
+    {
+      "epoch": 5.11,
+      "learning_rate": 0.00018847736625514404,
+      "loss": 0.3553,
+      "step": 41
+    },
+    {
+      "epoch": 5.22,
+      "learning_rate": 0.00018765432098765433,
+      "loss": 0.4037,
+      "step": 42
+    },
+    {
+      "epoch": 5.33,
+      "learning_rate": 0.00018683127572016462,
+      "loss": 0.3385,
+      "step": 43
+    },
+    {
+      "epoch": 5.44,
+      "learning_rate": 0.0001860082304526749,
+      "loss": 0.3168,
+      "step": 44
+    },
+    {
+      "epoch": 5.56,
+      "learning_rate": 0.0001851851851851852,
+      "loss": 0.3048,
+      "step": 45
+    },
+    {
+      "epoch": 5.67,
+      "learning_rate": 0.0001843621399176955,
+      "loss": 0.2698,
+      "step": 46
+    },
+    {
+      "epoch": 5.78,
+      "learning_rate": 0.00018353909465020578,
+      "loss": 0.2817,
+      "step": 47
+    },
+    {
+      "epoch": 5.89,
+      "learning_rate": 0.00018271604938271605,
+      "loss": 0.348,
+      "step": 48
+    },
+    {
+      "epoch": 6.11,
+      "learning_rate": 0.00018189300411522634,
+      "loss": 0.2683,
+      "step": 49
+    },
+    {
+      "epoch": 6.22,
+      "learning_rate": 0.00018106995884773663,
+      "loss": 0.2554,
+      "step": 50
+    },
+    {
+      "epoch": 6.33,
+      "learning_rate": 0.00018024691358024692,
+      "loss": 0.2441,
+      "step": 51
+    },
+    {
+      "epoch": 6.44,
+      "learning_rate": 0.0001794238683127572,
+      "loss": 0.2458,
+      "step": 52
+    },
+    {
+      "epoch": 6.56,
+      "learning_rate": 0.0001786008230452675,
+      "loss": 0.2161,
+      "step": 53
+    },
+    {
+      "epoch": 6.67,
+      "learning_rate": 0.00017777777777777779,
+      "loss": 0.1735,
+      "step": 54
+    },
+    {
+      "epoch": 6.78,
+      "learning_rate": 0.00017695473251028808,
+      "loss": 0.2945,
+      "step": 55
+    },
+    {
+      "epoch": 6.89,
+      "learning_rate": 0.00017613168724279837,
+      "loss": 0.2725,
+      "step": 56
+    },
+    {
+      "epoch": 7.11,
+      "learning_rate": 0.00017530864197530866,
+      "loss": 0.1834,
+      "step": 57
+    },
+    {
+      "epoch": 7.22,
+      "learning_rate": 0.00017448559670781895,
+      "loss": 0.2,
+      "step": 58
+    },
+    {
+      "epoch": 7.33,
+      "learning_rate": 0.00017366255144032924,
+      "loss": 0.1749,
+      "step": 59
+    },
+    {
+      "epoch": 7.44,
+      "learning_rate": 0.0001728395061728395,
+      "loss": 0.2132,
+      "step": 60
+    },
+    {
+      "epoch": 7.56,
+      "learning_rate": 0.0001720164609053498,
+      "loss": 0.1846,
+      "step": 61
+    },
+    {
+      "epoch": 7.67,
+      "learning_rate": 0.00017119341563786008,
+      "loss": 0.1885,
+      "step": 62
+    },
+    {
+      "epoch": 7.78,
+      "learning_rate": 0.00017037037037037037,
+      "loss": 0.1356,
+      "step": 63
+    },
+    {
+      "epoch": 7.89,
+      "learning_rate": 0.00016954732510288066,
+      "loss": 0.2781,
+      "step": 64
+    },
+    {
+      "epoch": 8.11,
+      "learning_rate": 0.00016872427983539098,
+      "loss": 0.1235,
+      "step": 65
+    },
+    {
+      "epoch": 8.22,
+      "learning_rate": 0.00016790123456790124,
+      "loss": 0.1649,
+      "step": 66
+    },
+    {
+      "epoch": 8.33,
+      "learning_rate": 0.00016707818930041153,
+      "loss": 0.1214,
+      "step": 67
+    },
+    {
+      "epoch": 8.44,
+      "learning_rate": 0.00016625514403292182,
+      "loss": 0.1199,
+      "step": 68
+    },
+    {
+      "epoch": 8.56,
+      "learning_rate": 0.0001654320987654321,
+      "loss": 0.1467,
+      "step": 69
+    },
+    {
+      "epoch": 8.67,
+      "learning_rate": 0.0001646090534979424,
+      "loss": 0.1244,
+      "step": 70
+    },
+    {
+      "epoch": 8.78,
+      "learning_rate": 0.0001637860082304527,
+      "loss": 0.1604,
+      "step": 71
+    },
+    {
+      "epoch": 8.89,
+      "learning_rate": 0.00016296296296296295,
+      "loss": 0.2041,
+      "step": 72
+    },
+    {
+      "epoch": 9.11,
+      "learning_rate": 0.00016213991769547324,
+      "loss": 0.1498,
+      "step": 73
+    },
+    {
+      "epoch": 9.22,
+      "learning_rate": 0.00016131687242798353,
+      "loss": 0.1273,
+      "step": 74
+    },
+    {
+      "epoch": 9.33,
+      "learning_rate": 0.00016049382716049385,
+      "loss": 0.1029,
+      "step": 75
+    },
+    {
+      "epoch": 9.44,
+      "learning_rate": 0.00015967078189300414,
+      "loss": 0.0845,
+      "step": 76
+    },
+    {
+      "epoch": 9.56,
+      "learning_rate": 0.00015884773662551443,
+      "loss": 0.0956,
+      "step": 77
+    },
+    {
+      "epoch": 9.67,
+      "learning_rate": 0.0001580246913580247,
+      "loss": 0.1048,
+      "step": 78
+    },
+    {
+      "epoch": 9.78,
+      "learning_rate": 0.00015720164609053498,
+      "loss": 0.1048,
+      "step": 79
+    },
+    {
+      "epoch": 9.89,
+      "learning_rate": 0.00015637860082304527,
+      "loss": 0.0788,
+      "step": 80
+    },
+    {
+      "epoch": 10.11,
+      "learning_rate": 0.00015555555555555556,
+      "loss": 0.0767,
+      "step": 81
+    },
+    {
+      "epoch": 10.22,
+      "learning_rate": 0.00015473251028806585,
+      "loss": 0.0862,
+      "step": 82
+    },
+    {
+      "epoch": 10.33,
+      "learning_rate": 0.00015390946502057614,
+      "loss": 0.0808,
+      "step": 83
+    },
+    {
+      "epoch": 10.44,
+      "learning_rate": 0.0001530864197530864,
+      "loss": 0.0923,
+      "step": 84
+    },
+    {
+      "epoch": 10.56,
+      "learning_rate": 0.00015226337448559672,
+      "loss": 0.0678,
+      "step": 85
+    },
+    {
+      "epoch": 10.67,
+      "learning_rate": 0.000151440329218107,
+      "loss": 0.0749,
+      "step": 86
+    },
+    {
+      "epoch": 10.78,
+      "learning_rate": 0.0001506172839506173,
+      "loss": 0.0934,
+      "step": 87
+    },
+    {
+      "epoch": 10.89,
+      "learning_rate": 0.0001497942386831276,
+      "loss": 0.0814,
+      "step": 88
+    },
+    {
+      "epoch": 11.11,
+      "learning_rate": 0.00014897119341563788,
+      "loss": 0.0649,
+      "step": 89
+    },
+    {
+      "epoch": 11.22,
+      "learning_rate": 0.00014814814814814815,
+      "loss": 0.0611,
+      "step": 90
+    },
+    {
+      "epoch": 11.33,
+      "learning_rate": 0.00014732510288065844,
+      "loss": 0.0566,
+      "step": 91
+    },
+    {
+      "epoch": 11.44,
+      "learning_rate": 0.00014650205761316873,
+      "loss": 0.0534,
+      "step": 92
+    },
+    {
+      "epoch": 11.56,
+      "learning_rate": 0.00014567901234567902,
+      "loss": 0.0645,
+      "step": 93
+    },
+    {
+      "epoch": 11.67,
+      "learning_rate": 0.0001448559670781893,
+      "loss": 0.0692,
+      "step": 94
+    },
+    {
+      "epoch": 11.78,
+      "learning_rate": 0.0001440329218106996,
+      "loss": 0.0605,
+      "step": 95
+    },
+    {
+      "epoch": 11.89,
+      "learning_rate": 0.00014320987654320989,
+      "loss": 0.0865,
+      "step": 96
+    },
+    {
+      "epoch": 12.11,
+      "learning_rate": 0.00014238683127572018,
+      "loss": 0.0372,
+      "step": 97
+    },
+    {
+      "epoch": 12.22,
+      "learning_rate": 0.00014156378600823047,
+      "loss": 0.0466,
+      "step": 98
+    },
+    {
+      "epoch": 12.33,
+      "learning_rate": 0.00014074074074074076,
+      "loss": 0.0525,
+      "step": 99
+    },
+    {
+      "epoch": 12.44,
+      "learning_rate": 0.00013991769547325105,
+      "loss": 0.0414,
+      "step": 100
+    },
+    {
+      "epoch": 12.56,
+      "learning_rate": 0.00013909465020576134,
+      "loss": 0.0586,
+      "step": 101
+    },
+    {
+      "epoch": 12.67,
+      "learning_rate": 0.0001382716049382716,
+      "loss": 0.0392,
+      "step": 102
+    },
+    {
+      "epoch": 12.78,
+      "learning_rate": 0.0001374485596707819,
+      "loss": 0.0449,
+      "step": 103
+    },
+    {
+      "epoch": 12.89,
+      "learning_rate": 0.00013662551440329218,
+      "loss": 0.0451,
+      "step": 104
+    },
+    {
+      "epoch": 13.11,
+      "learning_rate": 0.00013580246913580247,
+      "loss": 0.0352,
+      "step": 105
+    },
+    {
+      "epoch": 13.22,
+      "learning_rate": 0.00013497942386831276,
+      "loss": 0.0415,
+      "step": 106
+    },
+    {
+      "epoch": 13.33,
+      "learning_rate": 0.00013415637860082305,
+      "loss": 0.0348,
+      "step": 107
+    },
+    {
+      "epoch": 13.44,
+      "learning_rate": 0.00013333333333333334,
+      "loss": 0.0325,
+      "step": 108
+    },
+    {
+      "epoch": 13.56,
+      "learning_rate": 0.00013251028806584363,
+      "loss": 0.0383,
+      "step": 109
+    },
+    {
+      "epoch": 13.67,
+      "learning_rate": 0.00013168724279835392,
+      "loss": 0.041,
+      "step": 110
+    },
+    {
+      "epoch": 13.78,
+      "learning_rate": 0.0001308641975308642,
+      "loss": 0.0462,
+      "step": 111
+    },
+    {
+      "epoch": 13.89,
+      "learning_rate": 0.0001300411522633745,
+      "loss": 0.0278,
+      "step": 112
+    },
+    {
+      "epoch": 14.11,
+      "learning_rate": 0.00012921810699588476,
+      "loss": 0.0356,
+      "step": 113
+    },
+    {
+      "epoch": 14.22,
+      "learning_rate": 0.00012839506172839505,
+      "loss": 0.0419,
+      "step": 114
+    },
+    {
+      "epoch": 14.33,
+      "learning_rate": 0.00012757201646090534,
+      "loss": 0.0223,
+      "step": 115
+    },
+    {
+      "epoch": 14.44,
+      "learning_rate": 0.00012674897119341563,
+      "loss": 0.0291,
+      "step": 116
+    },
+    {
+      "epoch": 14.56,
+      "learning_rate": 0.00012592592592592592,
+      "loss": 0.0298,
+      "step": 117
+    },
+    {
+      "epoch": 14.67,
+      "learning_rate": 0.00012510288065843624,
+      "loss": 0.0244,
+      "step": 118
+    },
+    {
+      "epoch": 14.78,
+      "learning_rate": 0.0001242798353909465,
+      "loss": 0.0339,
+      "step": 119
+    },
+    {
+      "epoch": 14.89,
+      "learning_rate": 0.0001234567901234568,
+      "loss": 0.0305,
+      "step": 120
+    },
+    {
+      "epoch": 15.11,
+      "learning_rate": 0.00012263374485596708,
+      "loss": 0.0269,
+      "step": 121
+    },
+    {
+      "epoch": 15.22,
+      "learning_rate": 0.00012181069958847737,
+      "loss": 0.0292,
+      "step": 122
+    },
+    {
+      "epoch": 15.33,
+      "learning_rate": 0.00012098765432098766,
+      "loss": 0.0317,
+      "step": 123
+    },
+    {
+      "epoch": 15.44,
+      "learning_rate": 0.00012016460905349795,
+      "loss": 0.0303,
+      "step": 124
+    },
+    {
+      "epoch": 15.56,
+      "learning_rate": 0.00011934156378600823,
+      "loss": 0.0252,
+      "step": 125
+    },
+    {
+      "epoch": 15.67,
+      "learning_rate": 0.00011851851851851852,
+      "loss": 0.0325,
+      "step": 126
+    },
+    {
+      "epoch": 15.78,
+      "learning_rate": 0.00011769547325102881,
+      "loss": 0.0215,
+      "step": 127
+    },
+    {
+      "epoch": 15.89,
+      "learning_rate": 0.0001168724279835391,
+      "loss": 0.0247,
+      "step": 128
+    },
+    {
+      "epoch": 16.11,
+      "learning_rate": 0.00011604938271604939,
+      "loss": 0.0254,
+      "step": 129
+    },
+    {
+      "epoch": 16.22,
+      "learning_rate": 0.00011522633744855968,
+      "loss": 0.0274,
+      "step": 130
+    },
+    {
+      "epoch": 16.33,
+      "learning_rate": 0.00011440329218106996,
+      "loss": 0.0198,
+      "step": 131
+    },
+    {
+      "epoch": 16.44,
+      "learning_rate": 0.00011358024691358025,
+      "loss": 0.0254,
+      "step": 132
+    },
+    {
+      "epoch": 16.56,
+      "learning_rate": 0.00011275720164609054,
+      "loss": 0.0302,
+      "step": 133
+    },
+    {
+      "epoch": 16.67,
+      "learning_rate": 0.00011193415637860083,
+      "loss": 0.016,
+      "step": 134
+    },
+    {
+      "epoch": 16.78,
+      "learning_rate": 0.00011111111111111112,
+      "loss": 0.0224,
+      "step": 135
+    },
+    {
+      "epoch": 16.89,
+      "learning_rate": 0.0001102880658436214,
+      "loss": 0.0124,
+      "step": 136
+    },
+    {
+      "epoch": 17.11,
+      "learning_rate": 0.00010946502057613168,
+      "loss": 0.0204,
+      "step": 137
+    },
+    {
+      "epoch": 17.22,
+      "learning_rate": 0.00010864197530864197,
+      "loss": 0.026,
+      "step": 138
+    },
+    {
+      "epoch": 17.33,
+      "learning_rate": 0.00010781893004115226,
+      "loss": 0.0206,
+      "step": 139
+    },
+    {
+      "epoch": 17.44,
+      "learning_rate": 0.00010699588477366255,
+      "loss": 0.0151,
+      "step": 140
+    },
+    {
+      "epoch": 17.56,
+      "learning_rate": 0.00010617283950617284,
+      "loss": 0.0195,
+      "step": 141
+    },
+    {
+      "epoch": 17.67,
+      "learning_rate": 0.00010534979423868315,
+      "loss": 0.0184,
+      "step": 142
+    },
+    {
+      "epoch": 17.78,
+      "learning_rate": 0.00010452674897119341,
+      "loss": 0.0239,
+      "step": 143
+    },
+    {
+      "epoch": 17.89,
+      "learning_rate": 0.0001037037037037037,
+      "loss": 0.0251,
+      "step": 144
+    },
+    {
+      "epoch": 18.11,
+      "learning_rate": 0.00010288065843621399,
+      "loss": 0.0113,
+      "step": 145
+    },
+    {
+      "epoch": 18.22,
+      "learning_rate": 0.00010205761316872428,
+      "loss": 0.0179,
+      "step": 146
+    },
+    {
+      "epoch": 18.33,
+      "learning_rate": 0.00010123456790123458,
+      "loss": 0.0291,
+      "step": 147
+    },
+    {
+      "epoch": 18.44,
+      "learning_rate": 0.00010041152263374487,
+      "loss": 0.0234,
+      "step": 148
+    },
+    {
+      "epoch": 18.56,
+      "learning_rate": 9.958847736625515e-05,
+      "loss": 0.0206,
+      "step": 149
+    },
+    {
+      "epoch": 18.67,
+      "learning_rate": 9.876543209876543e-05,
+      "loss": 0.0155,
+      "step": 150
+    },
+    {
+      "epoch": 18.78,
+      "learning_rate": 9.794238683127573e-05,
+      "loss": 0.0125,
+      "step": 151
+    },
+    {
+      "epoch": 18.89,
+      "learning_rate": 9.711934156378602e-05,
+      "loss": 0.0242,
+      "step": 152
+    },
+    {
+      "epoch": 19.11,
+      "learning_rate": 9.62962962962963e-05,
+      "loss": 0.0136,
+      "step": 153
+    },
+    {
+      "epoch": 19.22,
+      "learning_rate": 9.547325102880659e-05,
+      "loss": 0.0207,
+      "step": 154
+    },
+    {
+      "epoch": 19.33,
+      "learning_rate": 9.465020576131688e-05,
+      "loss": 0.016,
+      "step": 155
+    },
+    {
+      "epoch": 19.44,
+      "learning_rate": 9.382716049382717e-05,
+      "loss": 0.0161,
+      "step": 156
+    },
+    {
+      "epoch": 19.56,
+      "learning_rate": 9.300411522633746e-05,
+      "loss": 0.0162,
+      "step": 157
+    },
+    {
+      "epoch": 19.67,
+      "learning_rate": 9.218106995884775e-05,
+      "loss": 0.0176,
+      "step": 158
+    },
+    {
+      "epoch": 19.78,
+      "learning_rate": 9.135802469135802e-05,
+      "loss": 0.0166,
+      "step": 159
+    },
+    {
+      "epoch": 19.89,
+      "learning_rate": 9.053497942386831e-05,
+      "loss": 0.0265,
+      "step": 160
+    },
+    {
+      "epoch": 20.11,
+      "learning_rate": 8.97119341563786e-05,
+      "loss": 0.0189,
+      "step": 161
+    },
+    {
+      "epoch": 20.22,
+      "learning_rate": 8.888888888888889e-05,
+      "loss": 0.021,
+      "step": 162
+    },
+    {
+      "epoch": 20.33,
+      "learning_rate": 8.806584362139918e-05,
+      "loss": 0.0178,
+      "step": 163
+    },
+    {
+      "epoch": 20.44,
+      "learning_rate": 8.724279835390947e-05,
+      "loss": 0.0116,
+      "step": 164
+    },
+    {
+      "epoch": 20.56,
+      "learning_rate": 8.641975308641975e-05,
+      "loss": 0.0147,
+      "step": 165
+    },
+    {
+      "epoch": 20.67,
+      "learning_rate": 8.559670781893004e-05,
+      "loss": 0.0154,
+      "step": 166
+    },
+    {
+      "epoch": 20.78,
+      "learning_rate": 8.477366255144033e-05,
+      "loss": 0.0133,
+      "step": 167
+    },
+    {
+      "epoch": 20.89,
+      "learning_rate": 8.395061728395062e-05,
+      "loss": 0.0144,
+      "step": 168
+    },
+    {
+      "epoch": 21.11,
+      "learning_rate": 8.312757201646091e-05,
+      "loss": 0.0131,
+      "step": 169
+    },
+    {
+      "epoch": 21.22,
+      "learning_rate": 8.23045267489712e-05,
+      "loss": 0.011,
+      "step": 170
+    },
+    {
+      "epoch": 21.33,
+      "learning_rate": 8.148148148148148e-05,
+      "loss": 0.0134,
+      "step": 171
+    },
+    {
+      "epoch": 21.44,
+      "learning_rate": 8.065843621399177e-05,
+      "loss": 0.0114,
+      "step": 172
+    },
+    {
+      "epoch": 21.56,
+      "learning_rate": 7.983539094650207e-05,
+      "loss": 0.014,
+      "step": 173
+    },
+    {
+      "epoch": 21.67,
+      "learning_rate": 7.901234567901235e-05,
+      "loss": 0.0214,
+      "step": 174
+    },
+    {
+      "epoch": 21.78,
+      "learning_rate": 7.818930041152264e-05,
+      "loss": 0.0198,
+      "step": 175
+    },
+    {
+      "epoch": 21.89,
+      "learning_rate": 7.736625514403293e-05,
+      "loss": 0.0271,
+      "step": 176
+    },
+    {
+      "epoch": 22.11,
+      "learning_rate": 7.65432098765432e-05,
+      "loss": 0.0129,
+      "step": 177
+    },
+    {
+      "epoch": 22.22,
+      "learning_rate": 7.57201646090535e-05,
+      "loss": 0.0156,
+      "step": 178
+    },
+    {
+      "epoch": 22.33,
+      "learning_rate": 7.48971193415638e-05,
+      "loss": 0.0182,
+      "step": 179
+    },
+    {
+      "epoch": 22.44,
+      "learning_rate": 7.407407407407407e-05,
+      "loss": 0.0088,
+      "step": 180
+    },
+    {
+      "epoch": 22.56,
+      "learning_rate": 7.325102880658436e-05,
+      "loss": 0.0146,
+      "step": 181
+    },
+    {
+      "epoch": 22.67,
+      "learning_rate": 7.242798353909465e-05,
+      "loss": 0.0099,
+      "step": 182
+    },
+    {
+      "epoch": 22.78,
+      "learning_rate": 7.160493827160494e-05,
+      "loss": 0.0176,
+      "step": 183
+    },
+    {
+      "epoch": 22.89,
+      "learning_rate": 7.078189300411523e-05,
+      "loss": 0.0228,
+      "step": 184
+    },
+    {
+      "epoch": 23.11,
+      "learning_rate": 6.995884773662552e-05,
+      "loss": 0.0084,
+      "step": 185
+    },
+    {
+      "epoch": 23.22,
+      "learning_rate": 6.91358024691358e-05,
+      "loss": 0.0109,
+      "step": 186
+    },
+    {
+      "epoch": 23.33,
+      "learning_rate": 6.831275720164609e-05,
+      "loss": 0.0092,
+      "step": 187
+    },
+    {
+      "epoch": 23.44,
+      "learning_rate": 6.748971193415638e-05,
+      "loss": 0.0117,
+      "step": 188
+    },
+    {
+      "epoch": 23.56,
+      "learning_rate": 6.666666666666667e-05,
+      "loss": 0.0161,
+      "step": 189
+    },
+    {
+      "epoch": 23.67,
+      "learning_rate": 6.584362139917696e-05,
+      "loss": 0.0128,
+      "step": 190
+    },
+    {
+      "epoch": 23.78,
+      "learning_rate": 6.502057613168725e-05,
+      "loss": 0.0242,
+      "step": 191
+    },
+    {
+      "epoch": 23.89,
+      "learning_rate": 6.419753086419753e-05,
+      "loss": 0.0147,
+      "step": 192
+    },
+    {
+      "epoch": 24.11,
+      "learning_rate": 6.337448559670782e-05,
+      "loss": 0.0146,
+      "step": 193
+    },
+    {
+      "epoch": 24.22,
+      "learning_rate": 6.255144032921812e-05,
+      "loss": 0.009,
+      "step": 194
+    },
+    {
+      "epoch": 24.33,
+      "learning_rate": 6.17283950617284e-05,
+      "loss": 0.01,
+      "step": 195
+    },
+    {
+      "epoch": 24.44,
+      "learning_rate": 6.0905349794238687e-05,
+      "loss": 0.0151,
+      "step": 196
+    },
+    {
+      "epoch": 24.56,
+      "learning_rate": 6.0082304526748977e-05,
+      "loss": 0.0141,
+      "step": 197
+    },
+    {
+      "epoch": 24.67,
+      "learning_rate": 5.925925925925926e-05,
+      "loss": 0.0155,
+      "step": 198
+    },
+    {
+      "epoch": 24.78,
+      "learning_rate": 5.843621399176955e-05,
+      "loss": 0.01,
+      "step": 199
+    },
+    {
+      "epoch": 24.89,
+      "learning_rate": 5.761316872427984e-05,
+      "loss": 0.0119,
+      "step": 200
+    },
+    {
+      "epoch": 25.11,
+      "learning_rate": 5.679012345679012e-05,
+      "loss": 0.0092,
+      "step": 201
+    },
+    {
+      "epoch": 25.22,
+      "learning_rate": 5.596707818930041e-05,
+      "loss": 0.0119,
+      "step": 202
+    },
+    {
+      "epoch": 25.33,
+      "learning_rate": 5.51440329218107e-05,
+      "loss": 0.0147,
+      "step": 203
+    },
+    {
+      "epoch": 25.44,
+      "learning_rate": 5.4320987654320986e-05,
+      "loss": 0.0141,
+      "step": 204
+    },
+    {
+      "epoch": 25.56,
+      "learning_rate": 5.3497942386831277e-05,
+      "loss": 0.0145,
+      "step": 205
+    },
+    {
+      "epoch": 25.67,
+      "learning_rate": 5.267489711934157e-05,
+      "loss": 0.01,
+      "step": 206
+    },
+    {
+      "epoch": 25.78,
+      "learning_rate": 5.185185185185185e-05,
+      "loss": 0.0087,
+      "step": 207
+    },
+    {
+      "epoch": 25.89,
+      "learning_rate": 5.102880658436214e-05,
+      "loss": 0.025,
+      "step": 208
+    },
+    {
+      "epoch": 26.11,
+      "learning_rate": 5.020576131687244e-05,
+      "loss": 0.009,
+      "step": 209
+    },
+    {
+      "epoch": 26.22,
+      "learning_rate": 4.938271604938271e-05,
+      "loss": 0.0132,
+      "step": 210
+    },
+    {
+      "epoch": 26.33,
+      "learning_rate": 4.855967078189301e-05,
+      "loss": 0.0074,
+      "step": 211
+    },
+    {
+      "epoch": 26.44,
+      "learning_rate": 4.773662551440329e-05,
+      "loss": 0.0171,
+      "step": 212
+    },
+    {
+      "epoch": 26.56,
+      "learning_rate": 4.691358024691358e-05,
+      "loss": 0.0101,
+      "step": 213
+    },
+    {
+      "epoch": 26.67,
+      "learning_rate": 4.609053497942387e-05,
+      "loss": 0.0112,
+      "step": 214
+    },
+    {
+      "epoch": 26.78,
+      "learning_rate": 4.5267489711934157e-05,
+      "loss": 0.0138,
+      "step": 215
+    },
+    {
+      "epoch": 26.89,
+      "learning_rate": 4.4444444444444447e-05,
+      "loss": 0.0187,
+      "step": 216
+    },
+    {
+      "epoch": 27.11,
+      "learning_rate": 4.3621399176954737e-05,
+      "loss": 0.0119,
+      "step": 217
+    },
+    {
+      "epoch": 27.22,
+      "learning_rate": 4.279835390946502e-05,
+      "loss": 0.015,
+      "step": 218
+    },
+    {
+      "epoch": 27.33,
+      "learning_rate": 4.197530864197531e-05,
+      "loss": 0.0096,
+      "step": 219
+    },
+    {
+      "epoch": 27.44,
+      "learning_rate": 4.11522633744856e-05,
+      "loss": 0.0084,
+      "step": 220
+    },
+    {
+      "epoch": 27.56,
+      "learning_rate": 4.032921810699588e-05,
+      "loss": 0.0112,
+      "step": 221
+    },
+    {
+      "epoch": 27.67,
+      "learning_rate": 3.950617283950617e-05,
+      "loss": 0.016,
+      "step": 222
+    },
+    {
+      "epoch": 27.78,
+      "learning_rate": 3.868312757201646e-05,
+      "loss": 0.0078,
+      "step": 223
+    },
+    {
+      "epoch": 27.89,
+      "learning_rate": 3.786008230452675e-05,
+      "loss": 0.0129,
+      "step": 224
+    },
+    {
+      "epoch": 28.11,
+      "learning_rate": 3.7037037037037037e-05,
+      "loss": 0.0079,
+      "step": 225
+    },
+    {
+      "epoch": 28.22,
+      "learning_rate": 3.6213991769547327e-05,
+      "loss": 0.0115,
+      "step": 226
+    },
+    {
+      "epoch": 28.33,
+      "learning_rate": 3.539094650205762e-05,
+      "loss": 0.011,
+      "step": 227
+    },
+    {
+      "epoch": 28.44,
+      "learning_rate": 3.45679012345679e-05,
+      "loss": 0.0143,
+      "step": 228
+    },
+    {
+      "epoch": 28.56,
+      "learning_rate": 3.374485596707819e-05,
+      "loss": 0.0087,
+      "step": 229
+    },
+    {
+      "epoch": 28.67,
+      "learning_rate": 3.292181069958848e-05,
+      "loss": 0.0166,
+      "step": 230
+    },
+    {
+      "epoch": 28.78,
+      "learning_rate": 3.209876543209876e-05,
+      "loss": 0.0099,
+      "step": 231
+    },
+    {
+      "epoch": 28.89,
+      "learning_rate": 3.127572016460906e-05,
+      "loss": 0.009,
+      "step": 232
+    },
+    {
+      "epoch": 29.11,
+      "learning_rate": 3.0452674897119343e-05,
+      "loss": 0.0115,
+      "step": 233
+    },
+    {
+      "epoch": 29.22,
+      "learning_rate": 2.962962962962963e-05,
+      "loss": 0.0086,
+      "step": 234
+    },
+    {
+      "epoch": 29.33,
+      "learning_rate": 2.880658436213992e-05,
+      "loss": 0.0097,
+      "step": 235
+    },
+    {
+      "epoch": 29.44,
+      "learning_rate": 2.7983539094650207e-05,
+      "loss": 0.0094,
+      "step": 236
+    },
+    {
+      "epoch": 29.56,
+      "learning_rate": 2.7160493827160493e-05,
+      "loss": 0.016,
+      "step": 237
+    },
+    {
+      "epoch": 29.67,
+      "learning_rate": 2.6337448559670787e-05,
+      "loss": 0.0119,
+      "step": 238
+    },
+    {
+      "epoch": 29.78,
+      "learning_rate": 2.551440329218107e-05,
+      "loss": 0.0122,
+      "step": 239
+    },
+    {
+      "epoch": 29.89,
+      "learning_rate": 2.4691358024691357e-05,
+      "loss": 0.007,
+      "step": 240
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 270,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 30,
+  "save_steps": 500,
+  "total_flos": 2.0720473325568e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}