---
library_name: peft
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- trl
- sft
- generated_from_trainer
datasets:
- generator
model-index:
- name: code-bench-Mistral-7B-text-to-sql-v1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# code-bench-Mistral-7B-text-to-sql-v1

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the generator dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4298

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 3

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 0.8246        | 0.0619 | 10   | 0.7742          |
| 0.6351        | 0.1238 | 20   | 0.7373          |
| 0.573         | 0.1858 | 30   | 0.5941          |
| 0.5628        | 0.2477 | 40   | 0.5330          |
| 0.5169        | 0.3096 | 50   | 0.5149          |
| 0.5002        | 0.3715 | 60   | 0.4909          |
| 0.4825        | 0.4334 | 70   | 0.4721          |
| 0.466         | 0.4954 | 80   | 0.4628          |
| 0.4523        | 0.5573 | 90   | 0.4580          |
| 0.4536        | 0.6192 | 100  | 0.4474          |
| 0.4435        | 0.6811 | 110  | 0.4356          |
| 0.4399        | 0.7430 | 120  | 0.4318          |
| 0.4073        | 0.8050 | 130  | 0.4261          |
| 0.4372        | 0.8669 | 140  | 0.4198          |
| 0.4046        | 0.9288 | 150  | 0.4135          |
| 0.3836        | 0.9907 | 160  | 0.4131          |
| 0.2902        | 1.0526 | 170  | 0.4158          |
| 0.3228        | 1.1146 | 180  | 0.4141          |
| 0.3094        | 1.1765 | 190  | 0.4167          |
| 0.294         | 1.2384 | 200  | 0.4127          |
| 0.3081        | 1.3003 | 210  | 0.4047          |
| 0.285         | 1.3622 | 220  | 0.4034          |
| 0.2814        | 1.4241 | 230  | 0.4065          |
| 0.2649        | 1.4861 | 240  | 0.4016          |
| 0.2715        | 1.5480 | 250  | 0.3956          |
| 0.2715        | 1.6099 | 260  | 0.3967          |
| 0.2768        | 1.6718 | 270  | 0.3914          |
| 0.2618        | 1.7337 | 280  | 0.3905          |
| 0.2663        | 1.7957 | 290  | 0.3887          |
| 0.2639        | 1.8576 | 300  | 0.3844          |
| 0.2601        | 1.9195 | 310  | 0.3831          |
| 0.2625        | 1.9814 | 320  | 0.3808          |
| 0.1607        | 2.0433 | 330  | 0.4298          |
| 0.1505        | 2.1053 | 340  | 0.4149          |
| 0.1483        | 2.1672 | 350  | 0.4370          |
| 0.1597        | 2.2291 | 360  | 0.4287          |
| 0.1456        | 2.2910 | 370  | 0.4301          |
| 0.1567        | 2.3529 | 380  | 0.4304          |
| 0.1469        | 2.4149 | 390  | 0.4247          |
| 0.1539        | 2.4768 | 400  | 0.4291          |
| 0.1546        | 2.5387 | 410  | 0.4330          |
| 0.1434        | 2.6006 | 420  | 0.4297          |
| 0.1387        | 2.6625 | 430  | 0.4301          |
| 0.1491        | 2.7245 | 440  | 0.4302          |
| 0.1538        | 2.7864 | 450  | 0.4298          |
| 0.1445        | 2.8483 | 460  | 0.4299          |
| 0.1405        | 2.9102 | 470  | 0.4299          |
| 0.1392        | 2.9721 | 480  | 0.4298          |


### Framework versions

- PEFT 0.12.0
- Transformers 4.44.2
- Pytorch 2.5.1+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1