initial-commit

.gitattributes

@@ -33,4 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
 demo.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,525 @@
----
-license: cc-by-4.0
----
+---
+base_model:  snorbyte/snorTTS-Indic-v0
+tags:
+    - text-to-speech
+    - tts
+    - transformers
+    - unsloth
+    - llama
+    - audio
+    - speech-synthesis
+license: apache-2.0
+language:
+    - hi
+    - gu
+    - mr
+    - pa
+    - bn
+    - te
+    - kn
+    - ml
+    - ta
+---
+
+# snorTTS-Indic-v0
+
+**Open-source multilingual Indic TTS model**  
+Human Sounding Indic TTS multi-stage Finetuned by Snorbyte on 140 hrs of proprietary speech across 9 Indic languages. The Base model is a LLaMA-3.2-3B Instruct model pretrained in 100k hours of English and finetuned in Hindi by canopylabs.
+
+---
+
+## Capabilities
+
+- Human Sounding Speech
+- Natural Human-like Delivery of Colloquial Transcripts (with English Mix and disfluencies)
+- Multi-Lingual Code Switching
+
+---
+## Model Overview
+
+| Item                   | Details                                                                                                                    |
+|------------------------|----------------------------------------------------------------------------------------------------------------------------|
+| **Base model**         | `canopylabs/3b-hi-pretrain-research_release`                                                                               |
+| **Architecture**       | LLaMA-3.2-3B-Instruct (`transformers`)                                                                                     |
+| **Audio codec**        | SNAC @ 24 kHz, 3 codebooks (12,288 new tokens)                                                                             |
+| **Training toolkit**   | [Unsloth](https://github.com/unslothai/unsloth) + HF TRL                                                                   |
+| **Languages**          | Hindi (hi), Gujarati (gu), Marathi (mr), Punjabi (pa), Bengali (bn), Telugu (te), Kannada (kn), Malayalam (ml), Tamil (ta) |
+
+---
+## Inference 
+
+```python
+# Load SNAC Model
+snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
+logger.success("Loaded SNAC model for audio decoding.")
+```
+
+```python
+# Function to construct audio file from SNAC codes generated by Model
+def generate_audio(
+    row, model, user=False, temperature=0.4, top_p=0.9, repetition_penalty=1.05
+):
+    if user:
+        prompt = row["eval_text_user"]
+    else:
+        prompt = row["eval_text_no_user"]
+    inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
+    max_tokens = MAX_SEQ_LENGTH - inputs.input_ids.shape[1]
+    output = model.generate(
+        input_ids=inputs.input_ids.to("cuda"),
+        attention_mask=inputs.attention_mask.to("cuda"),
+        max_new_tokens=max_tokens,
+        temperature=temperature,
+        top_p=top_p,
+        repetition_penalty=repetition_penalty,
+        eos_token_id=end_of_speech_id,
+    )
+    audio_ids = []
+    for id in output[0]:
+        if id >= audio_start_id:
+            audio_ids.append(id.item())
+    clean_audio_ids = []
+    for i in range((len(audio_ids) + 1) // 7):
+        for j in range(7):
+            clean_audio_ids += [audio_ids[7 * i + j] - audio_start_id]
+    codes = [[], [], []]
+    for i in range((len(clean_audio_ids) + 1) // 7):
+        codes[0].append(clean_audio_ids[7 * i])
+        codes[1].append(clean_audio_ids[7 * i + 1] - 4096)
+        codes[2].append(clean_audio_ids[7 * i + 2] - (2 * 4096))
+        codes[2].append(clean_audio_ids[7 * i + 3] - (3 * 4096))
+        codes[1].append(clean_audio_ids[7 * i + 4] - (4 * 4096))
+        codes[2].append(clean_audio_ids[7 * i + 5] - (5 * 4096))
+        codes[2].append(clean_audio_ids[7 * i + 6] - (6 * 4096))
+    codes = [
+        torch.tensor(codes[0]).unsqueeze(0),
+        torch.tensor(codes[1]).unsqueeze(0),
+        torch.tensor(codes[2]).unsqueeze(0),
+    ]
+    try:
+        audio = snac_model.decode(codes)
+    except Exception as e:
+        logger.error(f"Error decoding audio: {e}")
+        return None
+    return audio.detach().squeeze().to("cpu").numpy()
+```
+
+```python
+prompt = {
+        "eval_text_no_user": f"<custom_token_3><|begin_of_text|>நிச்சயமா. ரோம் ல் இரவு நேரம் ரொம்ப அழகா இருக்கு—piazzaகள் சுத்துறதுக்கு நல்ல நேரம்.<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
+        },
+train_sample = generate_audio(prompt, model, True)
+if train_sample is None:
+    logger.error("Failed to generate audio")
+else:
+    sf.write("output.wav", train_sample, 24000)
+    logger.success("Generated and saved audio as output.wav")
+```
+---
+
+## Types of Prompts
+
+For better results, generate audio with specific speakerIds mentioned below.
+
+- **Normal prompt:** Just pass the transcript in the format below
+```python
+{
+    "eval_text_no_user": f"<custom_token_3><|begin_of_text|>நிச்சயமா. ரோம் ல் இரவு நேரம் ரொம்ப அழகா இருக்கு—piazzaகள் சுத்துறதுக்கு நல்ல நேரம்.<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
+},
+```
+- **Speaker specific prompt**: Stick to the same format, just pass ```<language>{speakerId}:`` before the transcript. You can make any speaker speak in any of the 9 Languages
+```python
+{
+    "eval_text_user": f"<custom_token_3><|begin_of_text|>hindi159:  चलते रहो इस सफर में बिना रुके, क्योंकि मंज़िलें खुद राह दिखाने लगती हैं <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
+}
+```
+---
+
+### Recommended Speaker Ids
+
+| Language  | Speakers     |
+|-----------|--------------|
+| Hindi     | [159,49]     |
+| Tamil     | [188,128]    |
+| Bengali   | [125]        |
+| Malayalam | [189,124]    |
+| Kannada   | [142,138]    |
+| Telugu    | [69,133]     |
+| Punjabi   | [191,67,201] |
+| Gujarati  | [62,190,187] |
+| Marathi   | [205,82]     |
+
+- **Multi-lingual transcript specific prompt**:. Stick to the same format, just pass ```<language>{speakerId}:`` before the transcript. Pass the native language of the speakerId. YOu can 
+```python
+{
+    "eval_text_user": f"<custom_token_3><|begin_of_text|>bengali125: मुझे तो लगा वो आएगा, ஆனா அவன் வந்து full drama பண்ணிட்டான், আর শেষে আবার আমাকে দোষ দিচ্ছে <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
+}
+```
+---
+
+## Training Details
+- **Dataset:** [indic-tts-sample-snac-encoded](https://huggingface.co/datasets/snorbyte/indic-tts-sample-snac-encoded) curated by [snorbyte](https://snorbyte.com)
+    - 135 hours (~68 k samples) split into:
+        - stage_1: Text-Reading (47 k scripted) + Semi-spontaneous dialogue (16 k)
+        - stage_2: Colloquial Conversational Snippets (4.4 k)
+        - eval: Evaluation samples for training (200)
+    - 9 Indic languages, balanced across high-/low-quality speakers.
+
+- **Hyperparameters:**
+    - LoRA rank: 192
+    - LoRA alpha: 384
+    - Learning rate
+    - Batch size
+    - Per Device Train Batch Size: 8
+    - Gradient Accumulation Steps: 4
+    - Optimizer: adamw_8bit
+    - Learning Rate: 2e-5 
+    - Scheduler: cosine
+    - Warmup Ratio: 0.02
+    - Epochs: 2
+    - Max Seq Length: 2048
+    - SFT Trainer Packing: True
+
+- **Compute**
+    - GPU: 1 NVIDIA H100 on Vast.ai
+---
+
+## Training Code
+
+```bash
+pip install torch unslot datasets loguru snac trl soundfile wandb transformers
+```
+
+```python
+from unsloth import FastLanguageModel
+
+import os
+
+from datasets import load_dataset
+from loguru import logger
+from snac import SNAC
+from trl import SFTConfig, SFTTrainer
+import soundfile as sf
+import torch
+import wandb
+from transformers import AutoModelForCausalLM
+from transformers import AutoTokenizer
+```
+
+```python
+# Set up constants and configurations.
+BASE_MODEL = "canopylabs/3b-hi-pretrain-research_release"
+STAGE = 1 #1 or 2 based on the dataset you are using
+if STAGE == 1:
+    TRAIN_CSV_PATH = "" #path to stage_1 csv dataset
+else:
+    TRAIN_CSV_PATH = "" #path to stage_2 csv dataset 
+
+MAX_SEQ_LENGTH = 2048
+PER_DEVICE_TRAIN_BATCH_SIZE = 8
+GRADIENT_ACCUMULATION_STEPS = 4
+HUGGINGFACE_TOKEN = "" #pass you huggingface token
+MODEL_NAME = "snorTTS-indic"
+WANDB_USERNAME = "" #pass your wandb username
+WANDB_PROJECT = "snorTTS-indic"
+WANDB_LOG_MODEL = "checkpoint"
+WANDB_RUN_NAME = "run-0"
+WANDB_RUN_ID = None
+SEED = 3407
+
+# Set up environment variables for Weights & Biases.
+os.environ["WANDB_PROJECT"] = WANDB_PROJECT
+os.environ["WANDB_LOG_MODEL"] = WANDB_LOG_MODEL
+```
+
+```python
+# Set up constants and configurations.
+BASE_MODEL = "canopylabs/3b-hi-pretrain-research_release"
+STAGE = 1 #1 or 2 based on the dataset you are using
+if STAGE == 1:
+    TRAIN_CSV_PATH = "" #path to stage_1 csv dataset
+else:
+    TRAIN_CSV_PATH = "" #path to stage_2 csv dataset 
+
+MAX_SEQ_LENGTH = 2048
+PER_DEVICE_TRAIN_BATCH_SIZE = 8
+GRADIENT_ACCUMULATION_STEPS = 4
+HUGGINGFACE_TOKEN = "" #pass you huggingface token
+MODEL_NAME = "snorTTS-indic"
+WANDB_USERNAME = "" #pass your wandb username
+WANDB_PROJECT = "snorTTS-indic"
+WANDB_LOG_MODEL = "checkpoint"
+WANDB_RUN_NAME = "run-0"
+WANDB_RUN_ID = None
+SEED = 3407
+
+# Set up environment variables for Weights & Biases.
+os.environ["WANDB_PROJECT"] = WANDB_PROJECT
+os.environ["WANDB_LOG_MODEL"] = WANDB_LOG_MODEL
+```
+
+```python
+# Load the model and tokenizer.
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name=BASE_MODEL,
+    load_in_4bit=true,
+    max_seq_length=MAX_SEQ_LENGTH,
+    token=HUGGINGFACE_TOKEN,
+)
+logger.success(f"Loaded model: {BASE_MODEL}")
+
+# Get parameter efficient fine-tuning model.
+model = FastLanguageModel.get_peft_model(
+    model,
+    r=192,
+    target_modules=[
+        "q_proj",
+        "k_proj",
+        "v_proj",
+        "o_proj",
+        "up_proj",
+        "down_proj",
+        "gate_proj",
+        "lm_head",
+        "embed_tokens",
+    ],
+    lora_alpha=384,
+    random_state=SEED,
+)
+
+# Load the special tokens for the tokenizer.
+tokeniser_length = 128256
+
+start_of_text_id = 128000
+end_of_text_id = 128009
+start_of_speech_id = tokeniser_length + 1
+end_of_speech_id = tokeniser_length + 2
+start_of_human_id = tokeniser_length + 3
+end_of_human_id = tokeniser_length + 4
+start_of_ai_id = tokeniser_length + 5
+end_of_ai_id = tokeniser_length + 6
+pad_token_id = tokeniser_length + 7
+audio_start_id = tokeniser_length + 10
+
+start_of_text_token = tokenizer.decode([start_of_text_id])
+end_of_text_token = tokenizer.decode([end_of_text_id])
+start_of_speech_token = tokenizer.decode([start_of_speech_id])
+end_of_speech_token = tokenizer.decode([end_of_speech_id])
+start_of_human_token = tokenizer.decode([start_of_human_id])
+end_of_human_token = tokenizer.decode([end_of_human_id])
+start_of_ai_token = tokenizer.decode([start_of_ai_id])
+end_of_ai_token = tokenizer.decode([end_of_ai_id])
+pad_token = tokenizer.decode([pad_token_id])
+audio_start_token = tokenizer.decode([audio_start_id])
+
+logger.success("Load special tokens for the tokenizer.")
+
+# Set the padding token and padding side.
+tokenizer.pad_token = pad_token
+tokenizer.padding_side = "left"
+logger.success("Set padding token and padding side for the tokenizer.")
+```
+
+```python
+# Load training and validation datasets.
+train_dataset = load_dataset("csv", data_files=TRAIN_CSV_PATH)["train"]
+eval_dataset = load_dataset("csv", data_files=VALID_CSV_PATH)["train"]
+
+if TRAIN_NUM_SAMPLES:
+    train_dataset = train_dataset.shuffle(seed=SEED).select(
+        range(min(TRAIN_NUM_SAMPLES, len(train_dataset)))
+    )
+
+if EVAL_NUM_SAMPLES:
+    eval_dataset = eval_dataset.shuffle(seed=SEED).select(
+        range(min(EVAL_NUM_SAMPLES, len(eval_dataset)))
+    )
+
+logger.success(
+    f"Loaded datasets: {len(train_dataset)} training samples, {len(eval_dataset)} evaluation samples."
+)
+```
+
+```python
+# Flatten (interleave) and get SNAC token IDs from the audio codes.
+def flatten_and_get_audio_input_ids(row):
+    audio_codes = row["snac_codes"]
+    if isinstance(audio_codes, str):
+        audio_codes = eval(audio_codes)
+    snac_token_ids = []
+    for i in range(len(audio_codes[0])):
+        snac_token_ids.append(audio_codes[0][i] + 128266)
+        snac_token_ids.append(audio_codes[1][2 * i] + 128266 + 4096)
+        snac_token_ids.append(audio_codes[2][4 * i] + 128266 + (2 * 4096))
+        snac_token_ids.append(audio_codes[2][(4 * i) + 1] + 128266 + (3 * 4096))
+        snac_token_ids.append(audio_codes[1][(2 * i) + 1] + 128266 + (4 * 4096))
+        snac_token_ids.append(audio_codes[2][(4 * i) + 2] + 128266 + (5 * 4096))
+        snac_token_ids.append(audio_codes[2][(4 * i) + 3] + 128266 + (6 * 4096))
+    row["snac_token_ids"] = snac_token_ids
+    return row
+
+
+train_dataset = train_dataset.map(flatten_and_get_audio_input_ids)
+eval_dataset = eval_dataset.map(flatten_and_get_audio_input_ids)
+logger.success("Flattened and extracted SNAC token IDs from audio codes.")
+```
+
+```python
+# Filter out rows with empty or None audio codes.
+train_dataset = train_dataset.filter(
+    lambda x: x["snac_token_ids"] is not None and len(x["snac_token_ids"]) > 0
+)
+eval_dataset = eval_dataset.filter(
+    lambda x: x["snac_token_ids"] is not None and len(x["snac_token_ids"]) > 0
+)
+logger.success("Filtered datasets to remove rows with empty or None audio codes.")
+```
+
+```python
+# Remove duplicate frames from the audio codes.
+def remove_duplicate_frames(row):
+    vals = row["snac_token_ids"]
+    if len(vals) % 7 != 0:
+        raise ValueError("Input list length must be divisible by 7")
+    result = vals[:7]
+    for i in range(7, len(vals), 7):
+        current_first = vals[i]
+        previous_first = result[-7]
+        if current_first != previous_first:
+            result.extend(vals[i : i + 7])
+    row["snac_token_ids"] = result
+    return row
+
+
+train_dataset = train_dataset.map(remove_duplicate_frames)
+eval_dataset = eval_dataset.map(remove_duplicate_frames)
+logger.success("Removed duplicate frames from audio codes.")
+```
+
+```python
+# Define a function to format the prompt for each row in the dataset.
+def format_text(row):
+    text = (
+        f"{start_of_human_token}{start_of_text_token}{row['language']}{row['user']}: {row['utterance']}{end_of_text_token}"
+        f"{end_of_human_token}{start_of_ai_token}{start_of_speech_token}"
+        f"{tokenizer.decode(row['snac_token_ids'])}{end_of_speech_token}{end_of_ai_token}"
+    )
+    eval_text_user = (
+        f"{start_of_human_token}{start_of_text_token}{row['language']}{row['user']}: {row['utterance']}{end_of_text_token}"
+        f"{end_of_human_token}{start_of_ai_token}{start_of_speech_token}"
+    )
+    eval_text_no_user = (
+        f"{start_of_human_token}{start_of_text_token}{row['utterance']}{end_of_text_token}"
+        f"{end_of_human_token}{start_of_ai_token}{start_of_speech_token}"
+    )
+    row["text"] = text
+    row["eval_text_user"] = eval_text_user
+    row["eval_text_no_user"] = eval_text_no_user
+    return row
+
+
+train_dataset = train_dataset.map(format_text)
+eval_dataset = eval_dataset.map(format_text)
+logger.success("Formatted text for training and evaluation datasets.")
+```
+
+```python
+# Tokenize the text in the datasets without adding special tokens.
+def tokenize_function(example):
+    return tokenizer(
+        example["text"],
+        add_special_tokens=False,
+        truncation=True,
+        max_length=MAX_SEQ_LENGTH,
+    )
+
+
+train_dataset = train_dataset.map(tokenize_function)
+eval_dataset = eval_dataset.map(tokenize_function)
+logger.success("Tokenized text in the datasets without adding special tokens.")
+```
+
+```python
+# Set training arguments.
+training_args = SFTConfig(
+    num_train_epochs=2,
+    per_device_train_batch_size=PER_DEVICE_TRAIN_BATCH_SIZE,
+    gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
+    optim="adamw_8bit",
+    learning_rate=2e-5,
+    lr_scheduler_type="cosine",
+    warmup_ratio=0.02,
+    do_eval=True,
+    eval_strategy="steps",
+    eval_steps=50,
+    logging_strategy="steps",
+    logging_steps=1,
+    save_strategy="no",
+    save_only_model=True,
+    # save_steps=250,
+    output_dir="outputs",
+    report_to="wandb",
+    run_name=WANDB_RUN_NAME,
+    seed=SEED,
+)
+
+# Initialize the SFTTrainer.
+trainer = SFTTrainer(
+    model=model,
+    tokenizer=tokenizer,
+    train_dataset=train_dataset,
+    eval_dataset=eval_dataset,
+    dataset_text_field="text",
+    max_seq_length=MAX_SEQ_LENGTH,
+    dataset_num_proc=2,
+    packing=True,
+    args=training_args,
+)
+
+logger.success("Initialized SFTTrainer with the specified configuration.")
+```
+
+```python
+# Start the training process.
+logger.info("Starting the training process...")
+
+run = wandb.init()
+
+if WANDB_RUN_ID:
+    logger.info(f"Resuming from Weights & Biases run ID: {WANDB_RUN_ID}")
+
+    artifact = run.use_artifact(
+        f"{WANDB_USERNAME}/{WANDB_PROJECT}/{WANDB_RUN_ID}", type="model"
+    )
+
+    artifact_dir = artifact.download()
+
+    trainer.train(resume_from_checkpoint=artifact_dir)
+else:
+    try:
+        logger.info("Attempting to resume training from the last checkpoint...")
+
+        trainer.train(resume_from_checkpoint=True)
+    except Exception as err:
+        trainer.train()
+
+# Finish the Weights & Biases run.
+wandb.finish()
+
+logger.success("Training completed successfully.")
+```
+---
+
+## Citation
+
+BibTeX:
+
+```bibtex
+@misc{indictextaudio2025,
+  title={snorTTS-Indic-v0: Multilingual Indic TTS},
+  author={snorbyte},
+  year={2025},
+  howpublished={\url{snorbyte/snorTTS-Indic-v0}},
+  note={Apache-2.0}
+}
+```

chat_template.jinja

@@ -0,0 +1,93 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- if strftime_now is defined %}
+        {%- set date_string = strftime_now("%d %b %Y") %}
+    {%- else %}
+        {%- set date_string = "26 Jul 2024" %}
+    {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+        {{- '{"name": "' + tool_call.name + '", ' }}
+        {{- '"parameters": ' }}
+        {{- tool_call.arguments | tojson }}
+        {{- "}" }}
+        {{- "<|eot_id|>" }}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,37 @@
+{
+    "architectures": [
+        "LlamaForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 128000,
+    "eos_token_id": 128001,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 3072,
+    "initializer_range": 0.02,
+    "intermediate_size": 8192,
+    "max_position_embeddings": 131072,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "num_attention_heads": 24,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 8,
+    "pad_token_id": 128004,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-05,
+    "rope_scaling": {
+        "factor": 32.0,
+        "high_freq_factor": 4.0,
+        "low_freq_factor": 1.0,
+        "original_max_position_embeddings": 8192,
+        "rope_type": "llama3"
+    },
+    "rope_theta": 500000.0,
+    "tie_word_embeddings": true,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.53.1",
+    "unsloth_version": "2025.6.12",
+    "use_cache": true,
+    "vocab_size": 156940
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 128000,
+  "do_sample": true,
+  "eos_token_id": 128001,
+  "max_length": 131072,
+  "pad_token_id": 128004,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.53.1"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3df5ddcdf658979751c511457760629635b59a74268a968c3f984da836a75b2
+size 3438608176

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eec4f6982f07ae393c3b8891190530de25916db7c8ba0a6e4c91d47e3afe480b
+size 2466558064

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2157d2c55bd9948478f99016bd6b2cee9770ca1d950ba8924e9f2aea5161c836
+size 1661186464

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b999ed243b570750bc19b4fd3d513664e8552a38d79ed29439dfee7ecb346a36
+size 1928478848

model.safetensors.index.json

@@ -0,0 +1,262 @@
+{
+  "metadata": {
+    "total_size": 15131947008
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.norm.weight": "model-00003-of-00004.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,20 @@
+{
+  "additional_special_tokens": [
+    "<|audio|>"
+  ],
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|eot_id|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<custom_token_7>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:044e2a10201774018db120391980464472baabf223bd353cea49b17da0b66abc
+size 22849546