Instructions to use ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128")
model = AutoModelForCausalLM.from_pretrained("ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128

SGLang

How to use ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128 with Docker Model Runner:
```
docker model run hf.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128
```

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

EfficientQAT

EfficientQAT is a novel quantization technical, which pushes the limitation of uniform (INT) quantization in an efficient manner. Due to the leverage of standard INT quantization, the quantized model of EfficientQAT can also be transferred into other formats, such as GPTQ, BitBLAS, etc.

In this repo, we provide three type checkpoints, one is EQAT, indicats the original checkpoints of EfficientQAT. The other two are GPTQ and BitBLAS respectively.

Model Zoo

We provide a number of prequantized EfficientQAT models as follows:

WikiText2 PPL is measured in 2048 context length.
Avg. Accuracy indicate the average accuracy in 5 zero-shot reasoning tasks (WinoGrande,PIQA,HellaSwag,Arc-Easy, Arc-Challenge) with lm-eval v0.4.2.
1GB = $10^9$ Bit
Hub Link: EQAT indicates the original checkpoints. We also transfer the checkpoints into GPTQ and BitBLAS formats, which can be loaded directly through GPTQModel. (PS: GPTQModel is a official bug-fixed repo of AutoGPTQ, which would be merged into AutoGPTQ in future.)

Model	Quantization	WikiText2 PPL	Avg. Accuracy	Model Size (GB)	Hub link
Llama-2-7B	fp16	5.47	64.86	13.2	-
Llama-2-7B	w4g128	5.53	64.27	3.7	EQAT\|GPTQ\|BitBLAS
Llama-2-7B	w3g128	5.81	64.02	3.1	EQAT
Llama-2-7B	w2g64	6.86	60.14	2.3	EQAT\|GPTQ\|BitBLAS
Llama-2-7B	w2g128	7.17	59.50	2.2	EQAT\|GPTQ\|BitBLAS
Llama-2-13B	fp16	4.88	67.81	25.4	-
Llama-2-13B	w4g128	4.93	67.52	6.8	EQAT\|GPTQ\|BitBLAS
Llama-2-13B	w3g128	5.12	67.28	5.6	EQAT
Llama-2-13B	w2g64	5.96	64.88	4.0	EQAT\|GPTQ\|BitBLAS
Llama-2-13B	w2g128	6.08	63.88	3.8	EQAT\|GPTQ\|BitBLAS
Llama-2-70B	fp16	3.32	72.41	131.6	-
Llama-2-70B	w4g128	3.39	72.62	35.8	EQAT\|GPTQ\|BitBLAS
Llama-2-70B	w3g128	3.61	71.76	29.1	EQAT
Llama-2-70B	w2g64	4.52	69.48	20.1	EQAT\|GPTQ\|BitBLAS
Llama-2-70B	w2g128	4.61	68.93	18.9	EQAT\|GPTQ\|BitBLAS
Llama-3-8B	fp16	6.14	68.58	13.0	-
Llama-3-8B	w4g128	6.47	68.43	5.4	EQAT\|GPTQ\|BitBLAS
Llama-3-8B	w3g128	7.09	67.35	4.7	EQAT
Llama-3-8B	w2g64	9.41	60.76	3.9	EQAT\|GPTQ\|BitBLAS
Llama-3-8B	w2g128	9.80	59.36	3.8	EQAT\|GPTQ\|BitBLAS
Llama-3-70B	fp16	2.85	75.33	137.8	-
Llama-3-70B	w4g128	3.17	74.57	38.9	EQAT\|GPTQ\|BitBLAS
Llama-3-70B	w3g128	4.19	72.42	32.2	EQAT
Llama-3-70B	w2g64	6.08	67.89	23.2	EQAT\|GPTQ
Llama-3-70B	w2g128	6.38	67.57	22.0	EQAT\|GPTQ\|BitBLAS
Llama-3-8B-Instruct	fp16	8.29	68.43	13.0	-
Llama-3-8B-Instruct	w4g128	7.93	68.39	5.4	EQAT\|GPTQ\|BitBLAS
Llama-3-8B-Instruct	w3g128	8.55	67.24	4.7	EQAT
Llama-3-8B-Instruct	w2g64	11.19	60.66	3.9	EQAT\|GPTQ\|BitBLAS
Llama-3-8B-Instruct	w2g128	11.73	60.16	3.8	EQAT\|GPTQ\|BitBLAS
Llama-3-70B-Instruct	fp16	5.33	73.78	137.8	-
Llama-3-70B-Instruct	w4g128	5.35	73.47	38.9	EQAT\|GPTQ\|BitBLAS
Llama-3-70B-Instruct	w3g128	5.65	72.87	32.2	EQAT
Llama-3-70B-Instruct	w2g64	7.86	67.64	23.2	EQAT\|GPTQ\|BitBLAS
Llama-3-70B-Instruct	w2g128	8.14	67.54	22.0	EQAT\|GPTQ\|BitBLAS

Usage of EQAT models

Please refer https://github.com/OpenGVLab/EfficientQAT for details.

Usage of GPTQ and BitBLAS models

Below is an example to inference with GPTQ or BitBLAS quantized formats.

from transformers import AutoTokenizer
from gptqmodel import GPTQModel

quant_dir = "ChenMnZ/Llama-2-7b-EfficientQAT-w2g128-GPTQ"
# quant_dir = "ChenMnZ/Llama-2-7b-EfficientQAT-w2g128-BitBLAS"
# or local path

tokenizer = AutoTokenizer.from_pretrained(quant_dir, use_fast=True)


# load quantized model to the first GPU
model = GPTQModel.from_quantized(quant_dir)

# inference with model.generate
print(tokenizer.decode(model.generate(**tokenizer("Model quantization is", return_tensors="pt").to(model.device))[0]))

Citation

If you found this work useful, please consider citing:

@article{efficientqat,
  title={EfficientQAT: Efficient Quantization-Aware Training for Large Language Models},
  author={Chen, Mengzhao and Shao, Wenqi and Xu, Peng and Wang, Jiahao and Gao, Peng and Zhang, Kaipeng and Qiao, Yu and Luo, Ping},
  journal={arXiv preprint arXiv:2407.11062},
  year={2024}
}

Downloads last month: 3

Safetensors

Model size

11B params

Tensor type

F16

I32

Collection including ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128

EfficientQAT

Collection

The official prequantized EfficientQAT models. • 29 items • Updated Jul 18, 2024 • 2

Paper for ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

Paper • 2407.11062 • Published Jul 10, 2024 • 10