Model Card for nunchaku-t5

This repository contains Nunchaku-quantized versions of T5-XXL, used to encode text prompt to the embeddings. It is used to reduce the memory footprint of the model.

Model Details

Model Description

• Developed by: Nunchaku Team
• Model type: text-generation
• License: apache-2.0
• Quantized from model: t5_v1_1_xxl

Model Files

• awq-int4-flux.1-t5xxl.safetensors: AWQ quantized W4A16 T5-XXL model for FLUX.1.

Model Sources

• Inference Engine: nunchaku
• Quantization Library: deepcompressor
• Paper: SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
• Demo: svdquant.mit.edu

Usage

• Diffusers Usage: See flux.1-dev-qencoder.py. Check our tutorial for more advanced usage.
• ComfyUI Usage: See nunchaku-flux.1-dev-qencoder.json.

Citation

@inproceedings{
  li2024svdquant,
  title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models},
  author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}
@inproceedings{
  lin2023awq,
  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},
  booktitle={MLSys},
  year={2024}
}