v0.1.0 sweep: requant with 5.0-BPW ceiling, fresh benchmark numbers

README.md

@@ -1,6 +1,6 @@
 ---
 library_name: mlx
-license: apache-2.0
+license: gemma
 pipeline_tag: text-generation
 base_model: google/gemma-4-e4b-it
 tags:
@@ -12,39 +12,32 @@ tags:
 - optiq
 - apple-silicon
 - text-generation
-- gemma4
+- gemma-4
 ---
 
-# gemma-4-e4b-it-OptiQ-4bit
+# mlx-community/gemma-4-e4b-it-OptiQ-4bit
 
-> Optimized for Apple Silicon with [mlx-optiq](https://mlx-optiq.pages.dev/) — sensitivity-aware mixed-precision quantization, reusable at inference, fine-tuning, and serving time.
+A 4-bit mixed-precision MLX quant produced by [mlx-optiq](https://mlx-optiq.com/), the sensitivity-aware quantization toolkit for Apple Silicon.
 
-This is a mixed-precision quantized version of [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it) in MLX format. Instead of uniform 4-bit across every layer, optiq measures each layer's sensitivity via KL divergence on calibration data and assigns **per-layer** bit-widths (some layers at 8-bit, the rest at 4-bit) at the same average bits-per-weight. Same size, higher quality.
+A 4-bit mixed-precision MLX quant of [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it). Per-layer bit-widths come from a KL-divergence sensitivity pass on the bundled [`optiq.jsonl`](https://mlx-optiq.com/blog/calibration-mix) five-domain calibration mix (prose · reasoning · code · agent · tool-call). Sensitive layers go to 8-bit; robust ones stay at 4-bit. The on-disk size is within ~5 % of a stock uniform 4-bit MLX quant.
 
-The `optiq_metadata.json` sidecar ships in the repo; it's what `mlx-optiq` reads to drive sensitivity-aware LoRA fine-tuning, mixed-precision KV serving, and hot-swap adapter routing.
-
-**Multimodal stripping (default):** the base is published as a multimodal foundation model, but this optiq release ships the language stack only. Vision/audio config + token metadata are dropped at conversion time so the artifact is smaller, leaves more RAM for KV cache + LoRA, and runs cleanly on lower-spec Apple Silicon with longer contexts. If you need the vision tower, re-convert from the base with:
-
-```bash
-optiq convert google/gemma-4-e4b-it --target-bpw 4.5 --keep-unused-modalities -o ./mmodel
-```
-
-
-## Quantization Details
+## Quantization details
 
 | Property | Value |
 |---|---|
-| Target BPW | 4.5 |
-| Achieved BPW | 4.50 |
-| Layers at 8-bit (sensitive) | 149 |
-| Layers at 4-bit (robust) | 444 |
-| Total quantized layers | 593 |
+| Predominant precision | 4-bit |
+| Layers at 8-bit (sensitive) | 155 |
+| Layers at 4-bit (robust) | 224 |
+| Total quantized layers | 379 |
 | Group size | 64 |
-| `model_type` | `gemma4_text` |
+| Calibration mix | `optiq.jsonl` (32 samples × 5 domains) |
+| Reference for sensitivity | bf16 (auto-resolved; falls back to uniform-4-bit if bf16 doesn't fit) |
+
+We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform-4-bit at the same disk size. Benchmark deltas are below.
 
 ## Usage
 
-### Basic (works with stock `mlx-lm`)
+Load it with `mlx-lm` and use it as usual:
 
 ```bash
 pip install mlx-lm
@@ -59,73 +52,43 @@ response = generate(
     prompt="Explain quantum computing in simple terms.",
     max_tokens=200,
 )
-print(response)
 ```
 
-### Unlock the full stack with `mlx-optiq`
-
-Installing [mlx-optiq](https://pypi.org/project/mlx-optiq/) turns this model from a static checkpoint into a deployment-ready base:
+For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters, sandboxed Python execution for agent workflows), install [`mlx-optiq`](https://mlx-optiq.com/):
 
 ```bash
 pip install mlx-optiq
 ```
 
-**Mixed-precision KV-cache serving** (+40–62% decode speedup at 64k context on Qwen3.5 2B/4B/9B vs fp16 KV on M3 Max):
-
-```bash
-# One-time per-layer KV sensitivity pass
-optiq kv-cache mlx-community/gemma-4-e4b-it-OptiQ-4bit --target-bits 4.5 -o ./kv_cache
-
-# OpenAI-compatible server on :8080
-optiq serve \
-    --kv-config ./kv_cache/kv_config.json \
-    --model mlx-community/gemma-4-e4b-it-OptiQ-4bit \
-    --max-tokens 32768 --temp 0.6 --top-p 0.95
-```
-
-**Sensitivity-aware LoRA fine-tuning** — layers optiq kept at 8-bit (more sensitive) get 2× the adapter rank of layers at 4-bit, at the same base budget:
-
-```bash
-optiq lora train mlx-community/gemma-4-e4b-it-OptiQ-4bit \
-    --data ./my_data \
-    --rank 8 --rank-scaling by_bits \
-    --iters 1000 -o ./my_adapter
-```
-
-**Hot-swap adapters** — mount N adapters on one base, switch per request without reloading the model (adapter id via HF repo or local path, auto-downloaded):
-
-```bash
-optiq serve \
-    --model mlx-community/gemma-4-e4b-it-OptiQ-4bit \
-    --adapter ./my_adapter
-```
-
-Full documentation: [mlx-optiq.pages.dev](https://mlx-optiq.pages.dev/)
+See the [Gemma-4 family guide](https://mlx-optiq.com/docs/gemma-4) on [mlx-optiq.com](https://mlx-optiq.com/) for sampling defaults, training recipes, and family-specific caveats.
 
 ## Benchmarks
 
-**GSM8K** (200 samples, 3-shot chain-of-thought):
+Five-metric suite that drives the [Capability Score](https://mlx-optiq.com/blog/eval-framework):
 
-| Model | GSM8K Accuracy |
+| Metric | Score |
 |---|---:|
-| **This (optiq mixed 4.5 BPW)** | **see [Results page](https://mlx-optiq.pages.dev/results.html)** |
-| Uniform 4-bit baseline | (documented on Results) |
-
-See [mlx-optiq.pages.dev/results](https://mlx-optiq.pages.dev/results.html) for full methodology and per-model numbers.
+| MMLU (5-shot, 1000 samples) | 58.8% |
+| GSM8K (1000 samples, 3-shot CoT) | 77.8% |
+| IFEval (full set, strict) | 70.6% |
+| IFEval (full set, loose) | 70.8% |
+| BFCL-V3 simple (200 single-turn calls) | 69.0% |
+| HumanEval (164 problems, pass@1) | 76.8% |
+| **Capability Score** (mean of the 5 benchmarks above) | **70.6** |
+| KL vs bf16 reference (mean / p95) | 0.2755 / 1.3460 |
+| On-disk size | 6.1 GB |
+
+The Capability Score is the simple unweighted mean of the five benchmarks. Every metric gets one equal vote. Disk size is reported next to it as an honest second axis instead of being folded into the score. See the [eval-framework writeup](https://mlx-optiq.com/blog/eval-framework) for the full methodology.
 
 ## Links
 
-- **Documentation:** https://mlx-optiq.pages.dev/
-- **PyPI:** https://pypi.org/project/mlx-optiq/
-- **Article:** [Not All Layers Are Equal](https://x.com/latent_node/status/2028412948167942334?s=20)
-- **Base model:** [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it)
-
-## Credits
-
-- **Quantization method:** [mlx-optiq](https://pypi.org/project/mlx-optiq/)
+- **Project website:** [mlx-optiq.com](https://mlx-optiq.com/)
+- **Gemma-4 family guide:** [mlx-optiq.com/docs/gemma-4](https://mlx-optiq.com/docs/gemma-4)
+- **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/)
+- **Calibration mix:** [mlx-optiq.com/blog/calibration-mix](https://mlx-optiq.com/blog/calibration-mix)
+- **Eval framework:** [mlx-optiq.com/blog/eval-framework](https://mlx-optiq.com/blog/eval-framework)
 - **Base model:** [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it)
-- **Runtime:** [MLX](https://github.com/ml-explore/mlx)
 
 ## License
 
-Apache 2.0 (inherits from base model).
+Gemma license (inherits from base model). See https://ai.google.dev/gemma/terms for the terms of use.

chat_template.jinja

@@ -1,9 +1,9 @@
-{%- macro format_parameters(properties, required) -%}
+{%- macro format_parameters(properties, required, filter_keys=false) -%}
     {%- set standard_keys = ['description', 'type', 'properties', 'required', 'nullable'] -%}
     {%- set ns = namespace(found_first=false) -%}
     {%- for key, value in properties | dictsort -%}
         {%- set add_comma = false -%}
-        {%- if key not in standard_keys -%}
+        {%- if not filter_keys or key not in standard_keys -%}
             {%- if ns.found_first %},{% endif -%}
             {%- set ns.found_first = true -%}
             {{ key }}:{
@@ -65,7 +65,7 @@
                 {%- elif value is mapping -%}
                     {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
                     properties:{
-                    {{- format_parameters(value, value['required'] | default([])) -}}
+                    {{- format_parameters(value, value['required'] | default([]), filter_keys=true) -}}
                     }
                 {%- endif -%}
                 {%- if value['required'] -%}
@@ -178,18 +178,21 @@
 {#- Handle System/Tool Definitions Block -#}
 {%- if (enable_thinking is defined and enable_thinking) or tools or messages[0]['role'] in ['system', 'developer'] -%}
     {{- '<|turn>system\n' -}}
-
     {#- Inject Thinking token at the very top of the FIRST system turn -#}
     {%- if enable_thinking is defined and enable_thinking -%}
         {{- '<|think|>\n' -}}
         {%- set ns.prev_message_type = 'think' -%}
     {%- endif -%}
-
     {%- if messages[0]['role'] in ['system', 'developer'] -%}
-        {{- messages[0]['content'] | trim -}}
+        {%- if messages[0]['content'] is string -%}
+            {{- messages[0]['content'] | trim -}}
+        {%- elif messages[0]['content'] is sequence -%}
+            {%- for item in messages[0]['content'] -%}
+                {{- item['text'] | trim + ' '-}}
+            {%- endfor -%}
+        {%- endif -%}
         {%- set loop_messages = messages[1:] -%}
     {%- endif -%}
-
     {%- if tools -%}
         {%- for tool in tools %}
             {{- '<|tool>' -}}
@@ -198,7 +201,6 @@
         {%- endfor %}
         {%- set ns.prev_message_type = 'tool' -%}
     {%- endif -%}
-
     {{- '<turn|>\n' -}}
 {%- endif %}
 
@@ -302,6 +304,7 @@
                 {%- endfor -%}
             {%- endif -%}
 
+            {%- set captured_content -%}
             {%- if message['content'] is string -%}
                 {%- if role == 'model' -%}
                     {{- strip_thinking(message['content']) -}}
@@ -328,10 +331,14 @@
                     {%- endif -%}
                 {%- endfor -%}
             {%- endif -%}
+            {%- endset -%}
+
+            {{- captured_content -}}
+            {%- set has_content = captured_content | trim | length > 0 -%}
 
         {%- if ns.prev_message_type == 'tool_call' and not ns_tr_out.flag -%}
             {{- '<|tool_response>' -}}
-        {%- elif not (ns_tr_out.flag and not message.get('content')) -%}
+        {%- elif not (ns_tr_out.flag and not has_content) -%}
             {{- '<turn|>\n' -}}
         {%- endif -%}
     {%- endif -%}

model-00001-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:21d0d431c44f39e600ef6dda30d4e69afdd27fbc80c8139b73b277b53fe6bb46
-size 3297104547
+oid sha256:0d239262a51b1795d1556ba0d0bdea955c126108d053249d2fd4c1e1584100fd
+size 3523881390

model-00002-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40dea78dff54032cffb29fef4fb5b6a29b84ce11447663eaaaff983731339af4
-size 3023979841
+oid sha256:da846c36ac065e0c8f558cd27286757490fc345b6613ac2789750c0d760f59a3
+size 3010217360

optiq_metadata.json

@@ -1,2387 +1,1533 @@
 {
   "method": "optiq_mixed_precision",
-  "target_bpw": 4.5,
-  "achieved_bpw": 4.4995443518886935,
-  "n_high_bits": 149,
-  "n_low_bits": 444,
+  "base_model": "google/gemma-4-e4b-it",
+  "reference": "bf16",
+  "target_bpw": 5.0,
+  "achieved_bpw": 5.110966482264888,
+  "n_high_bits": 155,
+  "n_low_bits": 224,
   "threshold": 0.0,
   "per_layer": {
-    "model.language_model.layers.0.self_attn.q_proj": {
-      "bits": 8,
+    "language_model.model.per_layer_model_projection": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.0.self_attn.k_proj": {
+    "language_model.model.layers.41.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.0.self_attn.v_proj": {
+    "language_model.model.layers.41.per_layer_input_gate": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.0.self_attn.o_proj": {
+    "language_model.model.layers.41.mlp.up_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.0.mlp.gate_proj": {
+    "language_model.model.layers.41.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.0.mlp.up_proj": {
+    "language_model.model.layers.41.mlp.gate_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.0.mlp.down_proj": {
+    "language_model.model.layers.41.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.0.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.41.self_attn.v_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.0.per_layer_projection": {
-      "bits": 8,
+    "language_model.model.layers.41.self_attn.k_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.1.self_attn.q_proj": {
+    "language_model.model.layers.41.self_attn.q_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.1.self_attn.k_proj": {
+    "language_model.model.layers.40.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.1.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.40.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.1.self_attn.o_proj": {
+    "language_model.model.layers.40.mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.40.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.1.mlp.gate_proj": {
+    "language_model.model.layers.40.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.1.mlp.up_proj": {
+    "language_model.model.layers.40.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.1.mlp.down_proj": {
-      "bits": 8,
+    "language_model.model.layers.40.self_attn.v_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.1.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.40.self_attn.k_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.1.per_layer_projection": {
-      "bits": 8,
+    "language_model.model.layers.40.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.self_attn.q_proj": {
+    "language_model.model.layers.39.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.2.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.39.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.39.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.self_attn.o_proj": {
+    "language_model.model.layers.39.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.2.mlp.gate_proj": {
-      "bits": 8,
+    "language_model.model.layers.39.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.mlp.up_proj": {
+    "language_model.model.layers.39.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.mlp.down_proj": {
-      "bits": 8,
+    "language_model.model.layers.39.self_attn.v_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.39.self_attn.k_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.2.per_layer_projection": {
+    "language_model.model.layers.38.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.3.self_attn.q_proj": {
+    "language_model.model.layers.38.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.3.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.38.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.3.self_attn.v_proj": {
+    "language_model.model.layers.38.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.3.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.38.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.3.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.38.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.3.mlp.up_proj": {
+    "language_model.model.layers.38.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.3.mlp.down_proj": {
+    "language_model.model.layers.38.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.3.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.38.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.3.per_layer_projection": {
+    "language_model.model.layers.37.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.4.self_attn.q_proj": {
+    "language_model.model.layers.37.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.37.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.self_attn.v_proj": {
+    "language_model.model.layers.37.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.4.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.37.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.mlp.gate_proj": {
-      "bits": 8,
+    "language_model.model.layers.37.self_attn.o_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.mlp.up_proj": {
+    "language_model.model.layers.37.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.mlp.down_proj": {
-      "bits": 8,
+    "language_model.model.layers.37.self_attn.k_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.37.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.4.per_layer_projection": {
+    "language_model.model.layers.36.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.5.self_attn.q_proj": {
-      "bits": 8,
+    "language_model.model.layers.36.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.36.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.self_attn.v_proj": {
+    "language_model.model.layers.36.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.5.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.36.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.mlp.gate_proj": {
+    "language_model.model.layers.36.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.mlp.up_proj": {
+    "language_model.model.layers.36.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.mlp.down_proj": {
+    "language_model.model.layers.36.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.36.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.5.per_layer_projection": {
+    "language_model.model.layers.35.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.6.self_attn.q_proj": {
+    "language_model.model.layers.35.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.6.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.35.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.6.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.35.mlp.down_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.6.self_attn.o_proj": {
+    "language_model.model.layers.35.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.6.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.35.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.6.mlp.up_proj": {
+    "language_model.model.layers.35.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.6.mlp.down_proj": {
+    "language_model.model.layers.35.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.6.per_layer_input_gate": {
+    "language_model.model.layers.35.self_attn.q_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.6.per_layer_projection": {
+    "language_model.model.layers.34.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.7.self_attn.q_proj": {
+    "language_model.model.layers.34.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.34.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.self_attn.v_proj": {
+    "language_model.model.layers.34.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.7.self_attn.o_proj": {
+    "language_model.model.layers.34.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.mlp.gate_proj": {
+    "language_model.model.layers.34.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.mlp.up_proj": {
+    "language_model.model.layers.34.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.mlp.down_proj": {
+    "language_model.model.layers.34.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.34.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.7.per_layer_projection": {
+    "language_model.model.layers.33.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.8.self_attn.q_proj": {
+    "language_model.model.layers.33.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.33.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.self_attn.v_proj": {
+    "language_model.model.layers.33.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.8.self_attn.o_proj": {
+    "language_model.model.layers.33.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.mlp.gate_proj": {
+    "language_model.model.layers.33.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.mlp.up_proj": {
+    "language_model.model.layers.33.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.mlp.down_proj": {
+    "language_model.model.layers.33.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.33.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.8.per_layer_projection": {
+    "language_model.model.layers.32.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.9.self_attn.q_proj": {
+    "language_model.model.layers.32.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.9.self_attn.k_proj": {
+    "language_model.model.layers.32.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.9.self_attn.v_proj": {
+    "language_model.model.layers.32.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.9.self_attn.o_proj": {
+    "language_model.model.layers.32.mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.9.mlp.gate_proj": {
+    "language_model.model.layers.32.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.9.mlp.up_proj": {
+    "language_model.model.layers.32.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.9.mlp.down_proj": {
+    "language_model.model.layers.32.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.9.per_layer_input_gate": {
+    "language_model.model.layers.31.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.9.per_layer_projection": {
-      "bits": 8,
+    "language_model.model.layers.31.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.self_attn.q_proj": {
+    "language_model.model.layers.31.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.self_attn.k_proj": {
+    "language_model.model.layers.31.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.10.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.31.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.31.self_attn.o_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.mlp.gate_proj": {
-      "bits": 8,
+    "language_model.model.layers.31.self_attn.v_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.mlp.up_proj": {
+    "language_model.model.layers.31.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.mlp.down_proj": {
-      "bits": 8,
+    "language_model.model.layers.31.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.10.per_layer_input_gate": {
+    "language_model.model.layers.30.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.10.per_layer_projection": {
-      "bits": 8,
+    "language_model.model.layers.30.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.self_attn.q_proj": {
+    "language_model.model.layers.30.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.self_attn.k_proj": {
+    "language_model.model.layers.30.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.11.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.30.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.30.self_attn.o_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.mlp.gate_proj": {
+    "language_model.model.layers.30.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.mlp.up_proj": {
+    "language_model.model.layers.30.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.mlp.down_proj": {
+    "language_model.model.layers.30.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.11.per_layer_input_gate": {
+    "language_model.model.layers.29.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.11.per_layer_projection": {
-      "bits": 8,
+    "language_model.model.layers.29.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.12.self_attn.q_proj": {
+    "language_model.model.layers.29.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.12.self_attn.k_proj": {
+    "language_model.model.layers.29.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.12.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.29.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.12.self_attn.o_proj": {
+    "language_model.model.layers.29.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.12.mlp.gate_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.language_model.layers.12.mlp.up_proj": {
+    "language_model.model.layers.29.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.12.mlp.down_proj": {
+    "language_model.model.layers.29.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.12.per_layer_input_gate": {
+    "language_model.model.layers.29.self_attn.q_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.12.per_layer_projection": {
+    "language_model.model.layers.28.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.13.self_attn.q_proj": {
+    "language_model.model.layers.28.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.28.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.28.mlp.down_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.28.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.mlp.gate_proj": {
+    "language_model.model.layers.28.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.mlp.up_proj": {
+    "language_model.model.layers.28.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.mlp.down_proj": {
+    "language_model.model.layers.28.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.13.per_layer_input_gate": {
+    "language_model.model.layers.28.self_attn.q_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.13.per_layer_projection": {
+    "language_model.model.layers.27.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.14.self_attn.q_proj": {
+    "language_model.model.layers.27.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.27.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.self_attn.v_proj": {
+    "language_model.model.layers.27.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.14.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.27.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.mlp.gate_proj": {
+    "language_model.model.layers.27.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.mlp.up_proj": {
+    "language_model.model.layers.27.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.mlp.down_proj": {
+    "language_model.model.layers.27.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.27.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.14.per_layer_projection": {
+    "language_model.model.layers.26.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.15.self_attn.q_proj": {
+    "language_model.model.layers.26.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.15.self_attn.k_proj": {
+    "language_model.model.layers.26.mlp.up_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.15.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.26.mlp.down_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.15.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.26.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.15.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.26.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.15.mlp.up_proj": {
+    "language_model.model.layers.26.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.15.mlp.down_proj": {
+    "language_model.model.layers.26.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.15.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.26.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.15.per_layer_projection": {
+    "language_model.model.layers.25.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.16.self_attn.q_proj": {
-      "bits": 8,
+    "language_model.model.layers.25.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.25.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.self_attn.v_proj": {
+    "language_model.model.layers.25.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.16.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.25.mlp.gate_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.mlp.gate_proj": {
+    "language_model.model.layers.25.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.mlp.up_proj": {
+    "language_model.model.layers.25.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.mlp.down_proj": {
+    "language_model.model.layers.25.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.25.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.16.per_layer_projection": {
+    "language_model.model.layers.24.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.17.self_attn.q_proj": {
+    "language_model.model.layers.24.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.17.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.24.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.17.self_attn.v_proj": {
-      "bits": 8,
+    "language_model.model.layers.24.mlp.down_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.17.self_attn.o_proj": {
+    "language_model.model.layers.24.mlp.gate_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.17.mlp.gate_proj": {
+    "language_model.model.layers.24.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.17.mlp.up_proj": {
+    "language_model.model.layers.24.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.17.mlp.down_proj": {
+    "language_model.model.layers.24.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.17.per_layer_input_gate": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.17.per_layer_projection": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.18.self_attn.q_proj": {
+    "language_model.model.layers.24.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.18.self_attn.k_proj": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.18.self_attn.v_proj": {
+    "language_model.model.layers.23.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.18.self_attn.o_proj": {
+    "language_model.model.layers.23.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.18.mlp.gate_proj": {
+    "language_model.model.layers.23.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.18.mlp.up_proj": {
+    "language_model.model.layers.23.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.18.mlp.down_proj": {
+    "language_model.model.layers.23.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.18.per_layer_input_gate": {
+    "language_model.model.layers.23.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.18.per_layer_projection": {
+    "language_model.model.layers.23.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.19.self_attn.q_proj": {
-      "bits": 4,
+    "language_model.model.layers.23.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.19.self_attn.k_proj": {
+    "language_model.model.layers.23.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.19.self_attn.v_proj": {
+    "language_model.model.layers.22.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.19.self_attn.o_proj": {
+    "language_model.model.layers.22.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.19.mlp.gate_proj": {
+    "language_model.model.layers.22.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.19.mlp.up_proj": {
+    "language_model.model.layers.22.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.19.mlp.down_proj": {
+    "language_model.model.layers.22.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.19.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.22.self_attn.o_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.19.per_layer_projection": {
+    "language_model.model.layers.22.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.20.self_attn.q_proj": {
-      "bits": 4,
+    "language_model.model.layers.22.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.20.self_attn.k_proj": {
+    "language_model.model.layers.22.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.20.self_attn.v_proj": {
+    "language_model.model.layers.21.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.20.self_attn.o_proj": {
+    "language_model.model.layers.21.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.20.mlp.gate_proj": {
+    "language_model.model.layers.21.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.20.mlp.up_proj": {
+    "language_model.model.layers.21.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.20.mlp.down_proj": {
+    "language_model.model.layers.21.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.20.per_layer_input_gate": {
+    "language_model.model.layers.21.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.20.per_layer_projection": {
+    "language_model.model.layers.21.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.21.self_attn.q_proj": {
+    "language_model.model.layers.21.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.21.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.21.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.21.self_attn.v_proj": {
+    "language_model.model.layers.20.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.21.self_attn.o_proj": {
-      "bits": 8,
+    "language_model.model.layers.20.per_layer_input_gate": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.21.mlp.gate_proj": {
+    "language_model.model.layers.20.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.21.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.20.mlp.down_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.21.mlp.down_proj": {
+    "language_model.model.layers.20.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.21.per_layer_input_gate": {
+    "language_model.model.layers.20.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.21.per_layer_projection": {
+    "language_model.model.layers.20.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.22.self_attn.q_proj": {
+    "language_model.model.layers.20.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.22.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.20.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.22.self_attn.v_proj": {
+    "language_model.model.layers.19.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.22.self_attn.o_proj": {
+    "language_model.model.layers.19.per_layer_input_gate": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.22.mlp.gate_proj": {
+    "language_model.model.layers.19.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.22.mlp.up_proj": {
+    "language_model.model.layers.19.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.22.mlp.down_proj": {
+    "language_model.model.layers.19.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.22.per_layer_input_gate": {
+    "language_model.model.layers.19.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.22.per_layer_projection": {
+    "language_model.model.layers.19.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.23.self_attn.q_proj": {
+    "language_model.model.layers.19.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.23.self_attn.k_proj": {
-      "bits": 8,
+    "language_model.model.layers.19.self_attn.q_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.23.self_attn.v_proj": {
+    "language_model.model.layers.18.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.23.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.18.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.23.mlp.gate_proj": {
+    "language_model.model.layers.18.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.23.mlp.up_proj": {
+    "language_model.model.layers.18.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.23.mlp.down_proj": {
+    "language_model.model.layers.18.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.23.per_layer_input_gate": {
+    "language_model.model.layers.18.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.23.per_layer_projection": {
+    "language_model.model.layers.18.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.24.self_attn.q_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.language_model.layers.24.self_attn.o_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.language_model.layers.24.mlp.gate_proj": {
+    "language_model.model.layers.18.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.24.mlp.up_proj": {
+    "language_model.model.layers.18.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.24.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.17.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.24.per_layer_input_gate": {
+    "language_model.model.layers.17.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.24.per_layer_projection": {
+    "language_model.model.layers.17.mlp.up_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.25.self_attn.q_proj": {
+    "language_model.model.layers.17.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.25.self_attn.o_proj": {
+    "language_model.model.layers.17.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.25.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.17.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.25.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.17.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.25.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.17.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.25.per_layer_input_gate": {
+    "language_model.model.layers.17.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.25.per_layer_projection": {
+    "language_model.model.layers.16.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.26.self_attn.q_proj": {
-      "bits": 4,
+    "language_model.model.layers.16.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.26.self_attn.o_proj": {
+    "language_model.model.layers.16.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.26.mlp.gate_proj": {
+    "language_model.model.layers.16.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.26.mlp.up_proj": {
+    "language_model.model.layers.16.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.26.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.16.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.26.per_layer_input_gate": {
-      "bits": 4,
+    "language_model.model.layers.16.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.26.per_layer_projection": {
+    "language_model.model.layers.16.self_attn.k_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.27.self_attn.q_proj": {
+    "language_model.model.layers.16.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.27.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.15.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.27.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.15.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.27.mlp.up_proj": {
+    "language_model.model.layers.15.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.27.mlp.down_proj": {
+    "language_model.model.layers.15.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.27.per_layer_input_gate": {
+    "language_model.model.layers.15.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.27.per_layer_projection": {
+    "language_model.model.layers.15.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.28.self_attn.q_proj": {
-      "bits": 4,
+    "language_model.model.layers.15.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.28.self_attn.o_proj": {
+    "language_model.model.layers.15.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.28.mlp.gate_proj": {
+    "language_model.model.layers.15.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.28.mlp.up_proj": {
+    "language_model.model.layers.14.per_layer_projection": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.28.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.14.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.28.per_layer_input_gate": {
-      "bits": 8,
+    "language_model.model.layers.14.mlp.up_proj": {
+      "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.28.per_layer_projection": {
+    "language_model.model.layers.14.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.29.self_attn.q_proj": {
+    "language_model.model.layers.14.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.29.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.14.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.29.mlp.gate_proj": {
+    "language_model.model.layers.14.self_attn.v_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.29.mlp.up_proj": {
+    "language_model.model.layers.14.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.29.mlp.down_proj": {
+    "language_model.model.layers.14.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.29.per_layer_input_gate": {
-      "bits": 4,
+    "language_model.model.layers.13.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.29.per_layer_projection": {
+    "language_model.model.layers.13.per_layer_input_gate": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.30.self_attn.q_proj": {
+    "language_model.model.layers.13.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.30.self_attn.o_proj": {
+    "language_model.model.layers.13.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.30.mlp.gate_proj": {
+    "language_model.model.layers.13.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.30.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.13.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.30.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.13.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.30.per_layer_input_gate": {
+    "language_model.model.layers.13.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.30.per_layer_projection": {
+    "language_model.model.layers.12.per_layer_projection": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.31.self_attn.q_proj": {
-      "bits": 4,
+    "language_model.model.layers.12.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.31.self_attn.o_proj": {
+    "language_model.model.layers.12.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.31.mlp.gate_proj": {
+    "language_model.model.layers.12.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.31.mlp.up_proj": {
+    "language_model.model.layers.12.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.31.mlp.down_proj": {
+    "language_model.model.layers.12.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.31.per_layer_input_gate": {
-      "bits": 4,
+    "language_model.model.layers.12.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.31.per_layer_projection": {
+    "language_model.model.layers.12.self_attn.k_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.32.self_attn.q_proj": {
+    "language_model.model.layers.12.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.32.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.11.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.32.mlp.gate_proj": {
+    "language_model.model.layers.11.per_layer_input_gate": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.32.mlp.up_proj": {
+    "language_model.model.layers.11.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.32.mlp.down_proj": {
+    "language_model.model.layers.11.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.32.per_layer_input_gate": {
+    "language_model.model.layers.11.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.32.per_layer_projection": {
+    "language_model.model.layers.11.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.33.self_attn.q_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.language_model.layers.33.self_attn.o_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.language_model.layers.33.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.11.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.33.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.11.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.33.mlp.down_proj": {
+    "language_model.model.layers.11.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.33.per_layer_input_gate": {
-      "bits": 4,
+    "language_model.model.layers.10.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.33.per_layer_projection": {
+    "language_model.model.layers.10.per_layer_input_gate": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.34.self_attn.q_proj": {
+    "language_model.model.layers.10.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.34.self_attn.o_proj": {
+    "language_model.model.layers.10.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.34.mlp.gate_proj": {
+    "language_model.model.layers.10.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.34.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.10.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.34.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.10.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.34.per_layer_input_gate": {
-      "bits": 4,
+    "language_model.model.layers.10.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.34.per_layer_projection": {
+    "language_model.model.layers.10.self_attn.q_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.35.self_attn.q_proj": {
-      "bits": 4,
+    "language_model.model.layers.9.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.35.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.9.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.35.mlp.gate_proj": {
+    "language_model.model.layers.9.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.35.mlp.up_proj": {
+    "language_model.model.layers.9.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.35.mlp.down_proj": {
+    "language_model.model.layers.9.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.35.per_layer_input_gate": {
+    "language_model.model.layers.9.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.35.per_layer_projection": {
+    "language_model.model.layers.9.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.36.self_attn.q_proj": {
+    "language_model.model.layers.9.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.36.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.9.self_attn.q_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.36.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.8.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.36.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.8.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.36.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.8.mlp.up_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.36.per_layer_input_gate": {
+    "language_model.model.layers.8.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.36.per_layer_projection": {
+    "language_model.model.layers.8.mlp.gate_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.37.self_attn.q_proj": {
+    "language_model.model.layers.8.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.37.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.8.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.37.mlp.gate_proj": {
+    "language_model.model.layers.8.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.37.mlp.up_proj": {
+    "language_model.model.layers.8.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.37.mlp.down_proj": {
-      "bits": 4,
+    "language_model.model.layers.7.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.37.per_layer_input_gate": {
+    "language_model.model.layers.7.per_layer_input_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.37.per_layer_projection": {
+    "language_model.model.layers.7.mlp.down_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.38.self_attn.q_proj": {
+    "language_model.model.layers.7.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.38.self_attn.o_proj": {
+    "language_model.model.layers.7.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.38.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.7.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.38.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.7.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.38.mlp.down_proj": {
+    "language_model.model.layers.7.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.38.per_layer_input_gate": {
+    "language_model.model.layers.6.per_layer_projection": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.38.per_layer_projection": {
+    "language_model.model.layers.6.per_layer_input_gate": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.39.self_attn.q_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.language_model.layers.39.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.6.mlp.up_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.39.mlp.gate_proj": {
+    "language_model.model.layers.6.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.39.mlp.up_proj": {
-      "bits": 4,
+    "language_model.model.layers.6.mlp.gate_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.39.mlp.down_proj": {
+    "language_model.model.layers.6.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.39.per_layer_input_gate": {
-      "bits": 4,
+    "language_model.model.layers.6.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.39.per_layer_projection": {
+    "language_model.model.layers.6.self_attn.k_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.40.self_attn.q_proj": {
+    "language_model.model.layers.6.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.40.self_attn.o_proj": {
-      "bits": 4,
+    "language_model.model.layers.5.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.40.mlp.gate_proj": {
-      "bits": 4,
+    "language_model.model.layers.5.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.40.mlp.up_proj": {
+    "language_model.model.layers.5.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.40.mlp.down_proj": {
+    "language_model.model.layers.5.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.40.per_layer_input_gate": {
+    "language_model.model.layers.5.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.language_model.layers.40.per_layer_projection": {
+    "language_model.model.layers.5.self_attn.o_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.41.self_attn.q_proj": {
+    "language_model.model.layers.5.self_attn.v_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.41.self_attn.o_proj": {
+    "language_model.model.layers.5.self_attn.k_proj": {
       "bits": 8,
       "group_size": 64
     },
-    "model.language_model.layers.41.mlp.gate_proj": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.41.mlp.up_proj": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.41.mlp.down_proj": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.41.per_layer_input_gate": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.layers.41.per_layer_projection": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.language_model.per_layer_model_projection": {
-      "bits": 8,
-      "group_size": 64
-    },
-    "model.vision_tower.patch_embedder.input_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.0.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.1.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.2.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.3.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.4.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.5.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.6.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.7.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.8.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.9.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.10.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.11.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.12.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.13.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.14.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.self_attn.o_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.mlp.gate_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.mlp.up_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.vision_tower.encoder.layers.15.mlp.down_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.embed_vision.embedding_projection": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.subsample_conv_projection.input_proj_linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.0.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.1.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.feed_forward1.ffw_layer_2.linear": {
+    "language_model.model.layers.5.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.2.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.2.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.3.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.4.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.5.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.6.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.self_attn.q_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.self_attn.k_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.self_attn.v_proj.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.self_attn.post.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.self_attn.relative_k_proj": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.lconv1d.linear_start.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.7.lconv1d.linear_end.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.8.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.8.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
-      "group_size": 64
-    },
-    "model.audio_tower.layers.8.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
+    "language_model.model.layers.4.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
+    "language_model.model.layers.4.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.self_attn.q_proj.linear": {
+    "language_model.model.layers.4.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.self_attn.k_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.4.mlp.down_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.self_attn.v_proj.linear": {
+    "language_model.model.layers.4.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.self_attn.post.linear": {
+    "language_model.model.layers.4.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.self_attn.relative_k_proj": {
-      "bits": 4,
+    "language_model.model.layers.4.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.lconv1d.linear_start.linear": {
-      "bits": 4,
+    "language_model.model.layers.4.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.8.lconv1d.linear_end.linear": {
-      "bits": 4,
+    "language_model.model.layers.4.self_attn.q_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.feed_forward1.ffw_layer_1.linear": {
-      "bits": 4,
+    "language_model.model.layers.3.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
+    "language_model.model.layers.3.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.feed_forward2.ffw_layer_1.linear": {
+    "language_model.model.layers.3.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
+    "language_model.model.layers.3.mlp.down_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.self_attn.q_proj.linear": {
+    "language_model.model.layers.3.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.self_attn.k_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.3.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.self_attn.v_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.3.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.self_attn.post.linear": {
+    "language_model.model.layers.3.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.self_attn.relative_k_proj": {
+    "language_model.model.layers.3.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.lconv1d.linear_start.linear": {
+    "language_model.model.layers.2.per_layer_projection": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.9.lconv1d.linear_end.linear": {
-      "bits": 4,
+    "language_model.model.layers.2.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.feed_forward1.ffw_layer_1.linear": {
+    "language_model.model.layers.2.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.feed_forward1.ffw_layer_2.linear": {
+    "language_model.model.layers.2.mlp.down_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
+    "language_model.model.layers.2.mlp.gate_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.feed_forward2.ffw_layer_2.linear": {
+    "language_model.model.layers.2.self_attn.o_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.self_attn.q_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.2.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.self_attn.k_proj.linear": {
+    "language_model.model.layers.2.self_attn.k_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.self_attn.v_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.2.self_attn.q_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.self_attn.post.linear": {
+    "language_model.model.layers.1.per_layer_projection": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.self_attn.relative_k_proj": {
-      "bits": 4,
+    "language_model.model.layers.1.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.lconv1d.linear_start.linear": {
+    "language_model.model.layers.1.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.10.lconv1d.linear_end.linear": {
-      "bits": 4,
+    "language_model.model.layers.1.mlp.down_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.feed_forward1.ffw_layer_1.linear": {
+    "language_model.model.layers.1.mlp.gate_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.feed_forward1.ffw_layer_2.linear": {
-      "bits": 4,
+    "language_model.model.layers.1.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.feed_forward2.ffw_layer_1.linear": {
-      "bits": 4,
+    "language_model.model.layers.1.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.feed_forward2.ffw_layer_2.linear": {
-      "bits": 4,
+    "language_model.model.layers.1.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.self_attn.q_proj.linear": {
+    "language_model.model.layers.1.self_attn.q_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.self_attn.k_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.0.per_layer_projection": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.self_attn.v_proj.linear": {
-      "bits": 4,
+    "language_model.model.layers.0.per_layer_input_gate": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.self_attn.post.linear": {
+    "language_model.model.layers.0.mlp.up_proj": {
       "bits": 4,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.self_attn.relative_k_proj": {
-      "bits": 4,
+    "language_model.model.layers.0.mlp.down_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.lconv1d.linear_start.linear": {
-      "bits": 4,
+    "language_model.model.layers.0.mlp.gate_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.layers.11.lconv1d.linear_end.linear": {
-      "bits": 4,
+    "language_model.model.layers.0.self_attn.o_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.audio_tower.output_proj": {
-      "bits": 4,
+    "language_model.model.layers.0.self_attn.v_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "model.embed_audio.embedding_projection": {
-      "bits": 4,
+    "language_model.model.layers.0.self_attn.k_proj": {
+      "bits": 8,
       "group_size": 64
     },
-    "lm_head": {
-      "bits": 4,
+    "language_model.model.layers.0.self_attn.q_proj": {
+      "bits": 8,
       "group_size": 64
     }
   },
   "post_processing": [
     {
-      "op": "strip_vision",
+      "op": "strip_multimodal_metadata",
       "architectures": {
         "from": [
           "Gemma4ForConditionalGeneration"
@@ -2390,6 +1536,7 @@
           "Gemma4ForCausalLM"
         ]
       },
+      "model_type": null,
       "flattened_text_config": false,
       "dropped_keys": [
         "audio_config",
@@ -2402,13 +1549,8 @@
         "image_token_id",
         "video_token_id",
         "vision_soft_tokens_per_image"
-      ]
-    },
-    {
-      "op": "strip_vision",
-      "architectures": null,
-      "flattened_text_config": true,
-      "dropped_keys": []
+      ],
+      "dropped_mrope_keys": []
     }
   ]
 }