groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16

Model Card: groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16

Model Overview

Model Name: groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16 Base Model: huihui-ai/Qwen3-8B-abliterated Description: This is a quantized version of the uncensored huihui-ai/Qwen3-8B-abliterated model, derived from Qwen/Qwen3-8B. The model has been quantized to GPTQ Int8 W8A16 for maximum inference speed on NVIDIA 3090 GPUs. Abliteration was performed using a novel, faster method to remove refusals, making this a proof-of-concept implementation for uncensored language model behavior.

Important Note: A newer version, huihui-ai/Huihui-Qwen3-8B-abliterated-v2, is available. Consider using the updated version for improved performance.

Quantization Details

Quantization Method: GPTQ Int8 W8A16

Purpose: Optimized for high-speed inference on NVIDIA 3090 GPUs, reducing memory footprint while maintaining performance.

Impact: Provides faster inference compared to the unquantized model, suitable for resource-constrained environments.

Model Size: 2.98B parameters

Tensor Types: I64, I32, F16

Usage

Using with vLLM

The model can be used with vLLM for efficient inference. Below is an example of how to set up and run the model using vLLM in Python:

from vllm import LLM, SamplingParams

Define model ID

MODEL_ID = "groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16"

Initialize the vLLM model

llm = LLM( model=MODEL_ID, dtype="bfloat16", # Use bfloat16 for compatibility with GPTQ quantization trust_remote_code=True, quantization="gptq", # Specify GPTQ quantization gpu_memory_utilization=0.9, # Adjust based on your GPU memory )

Define sampling parameters

sampling_params = SamplingParams( temperature=0.7, max_tokens=8192, stop=["/exit"], # Custom stop token for interactive loop )

Interactive chat loop

system_prompt = "You are a helpful assistant." messages = [{"role": "system", "content": system_prompt}]

while True: user_input = input("User: ").strip() if user_input.lower() == "/exit": print("Exiting chat.") break if user_input.lower() == "/clear": messages = [{"role": "system", "content": system_prompt}] print("Chat history cleared. Starting a new conversation.") continue if not user_input: print("Input cannot be empty. Please enter something.") continue

# Append user input to messages
messages.append({"role": "user", "content": user_input})

# Format prompt for vLLM
prompt = "\n".join([f"{msg['role']}: {msg['content']}" for msg in messages])

# Generate response
outputs = llm.generate([prompt], sampling_params)
response = outputs[0].outputs[0].text.strip()

# Print and append response
print(f"Assistant: {response}")
messages.append({"role": "assistant", "content": response})

Installation Requirements

To use the model with vLLM, ensure you have vLLM installed:

pip install vllm

Notes

The model is pre-quantized to GPTQ Int8 W8A16, so specify quantization="gptq" when initializing the LLM object.

Adjust gpu_memory_utilization based on your GPU's memory capacity to avoid out-of-memory errors.

The max_tokens parameter can be increased for longer responses, but this may impact performance.

The model is not deployed by any inference provider. For provider support, contact the repository maintainers at Hugging Face.

Performance

Pass Rate for Harmful Instructions

The pass rate measures the proportion of harmful instructions that do not trigger refusals, calculated as (total - triggered_total) / total. The test set is sourced from huihui-ai/harmbench_behaviors, evaluated using TestPassed.py.

Test Results:

Model: huihui-ai/Qwen3-8B-abliterated

Passed Total: 320/320

Passed Ratio: 1.00 (100.00%)

Comparison:

Model

Passed Total

Passed Ratio

Qwen3-8B

195/320

60.94%

Qwen3-8B-abliterated

320/320

100.00%

Note: The test provides a preliminary assessment. For comprehensive results, consider increasing the max_tokens value during evaluation.

Limitations

This model is a proof-of-concept with abliteration to remove refusals, which may lead to unpredictable behavior on certain inputs.

The quantization to GPTQ Int8 W8A16 may introduce minor performance trade-offs compared to the unquantized model, though optimized for speed.

Users should verify outputs for sensitive applications, as the model is uncensored and may generate harmful or inappropriate content.

References

Repository: groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16

Base Model: Qwen/Qwen3-8B

Abliteration Method: remove-refusals-with-transformers

Test Set: huihui-ai/harmbench_behaviors

Newer Version: huihui-ai/Huihui-Qwen3-8B-abliterated-v2