Create README.md

README.md

@@ -0,0 +1,33 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+inference: true
+widget:
+- text: Hello!
+  example_title: Hello world
+  group: Python
+---
+
+This model is for debugging. It is randomly initialized using the config from [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) but with smaller size. 
+
+Codes:
+```python
+from awq import AutoAWQForCausalLM
+from transformers import AutoTokenizer
+
+model_path = "yujiepan/meta-llama-3.1-tiny-random-hidden128"
+quant_config = {
+    "zero_point": True,
+    "q_group_size": 64,
+    "w_bit": 4,
+    "version": "GEMM",
+}
+# Load model
+model = AutoAWQForCausalLM.from_pretrained(
+    model_path, low_cpu_mem_usage=True, use_cache=False, device_map='cuda',
+)
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+
+# Quantize
+model.quantize(tokenizer, quant_config=quant_config)
+```