|
|
--- |
|
|
library_name: exllamav3 |
|
|
license: apache-2.0 |
|
|
license_link: https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507/blob/main/LICENSE |
|
|
pipeline_tag: text-generation |
|
|
base_model: Qwen/Qwen3-235B-A22B-Instruct-2507 |
|
|
base_model_relation: quantized |
|
|
tags: |
|
|
- exl3 |
|
|
--- |
|
|
|
|
|
Exllamav3 quantization of [Qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) |
|
|
|
|
|
[2.25 bpw h6](https://huggingface.co/MikeRoz/Qwen3-235B-A22B-Instruct-2507-exl3/tree/2.25bpw_H6) 63.377 GiB |
|
|
[2.85 bpw h6](https://huggingface.co/MikeRoz/Qwen3-235B-A22B-Instruct-2507-exl3/tree/2.85bpw_H6) 79.714 GiB |
|
|
[3.00 bpw h6](https://huggingface.co/MikeRoz/Qwen3-235B-A22B-Instruct-2507-exl3/tree/3.00bpw_H6) 83.800 GiB |
|
|
[3.20 bpw h6](https://huggingface.co/MikeRoz/Qwen3-235B-A22B-Instruct-2507-exl3/tree/3.20bpw_H6) 89.177 GiB |
|
|
[3.60 bpw h6](https://huggingface.co/MikeRoz/Qwen3-235B-A22B-Instruct-2507-exl3/tree/3.60bpw_H6) 100.125 GiB |
|
|
[4.00 bpw h6](https://huggingface.co/MikeRoz/Qwen3-235B-A22B-Instruct-2507-exl3/tree/4.00bpw_H6) 111.013 GiB |
|
|
|
|
|
* The 2.25 bpw quant will fit in three 24 GB cards with 20k tokens of fp16 context. |
|
|
* The 2.85 bpw quant will fit in four 24 GB cards with 40k tokens of fp16 context. |
|
|
* The 3.60 bpw quant will fit in five 24 GB cards with 57k tokens of fp16 context. |
|
|
* The 4.00 bpw quant will fit in six 24 GB cards with 81,920 tokens of fp16 context. |
|
|
* The 3.20 bpw quant would not load at all on four 24 GB cards. I could only get the 3.00 to load with 4096 of q8 cache. These two are probably best left to five-card setups and above (or 96 GB on fewer cards). |
|
|
* Note that all these numbers are on the current version of exllamav3, which does not support tensor parallelism at this time. If you're reading this from a future where this feature has been implemented, or if you have larger cards, then you can probably do better than what I'm reporting here. |
|
|
|
