Updated ReadME.md

README.md

@@ -1 +1,145 @@
-First
+---
+language:
+- en
+library_name: vllm
+pipeline_tag: text-generation
+tags:
+  - text-generation
+  - conversational
+  - compressed-tensors
+  - awq
+  - w4a16
+  - int8
+  - quantized
+base_model: TheDrummer/Fallen-Command-A-111B-v1
+base_model_relation: quantized
+quantized_by: TheHouseOfTheDude
+---
+
+# Fallen-Command-A-111B-v1 — **Quantized** (compressed-tensors for vLLM)
+
+This repository provides **quantized runtime packages** of  
+**[TheDrummer/Fallen-Command-A-111B-v1](https://huggingface.co/TheDrummer/Fallen-Command-A-111B-v1)**, a finetune of  
+**[CohereLabs/c4ai-command-a-03-2025](https://huggingface.co/CohereLabs/c4ai-command-a-03-2025)** (aka **Command A**), repackaged for **vLLM** using the **compressed‑tensors** format.
+
+> **TL;DR**
+> - **This repo is quantized** with branches **W4A16** and **W8A16**.  
+> - Load with **vLLM** using `--quantization compressed-tensors`.  
+> - Command A (111B) is a **dense**, enterprise‑oriented model with **256K context**, high throughput, and strong capabilities for **tool use, agents, RAG, and multilingual** tasks.
+
+---
+
+## Revisions & Branches
+
+> The **`main`** branch is a **landing page** (model card + links). All runnable artifacts live under per‑revision branches.
+
+- **main** — placeholder / landing page  
+- **W4A16** — 4‑bit weights / 16‑bit activations builds and runtime assets  
+- **W8A16** — 8‑bit weights / 16‑bit activations builds
+
+---
+
+## Repository Contents (per revision)
+
+- **Sharded quantized weights** in `.safetensors` with an index (`model.safetensors.index.json`)  
+- `config.json` including **compressed‑tensors** metadata (`weight_format`, `quantization`, `quantization_config`)  
+- Tokenizer artifacts (`tokenizer.json`, `tokenizer.model`, etc.)  
+- Optional: `chat_template.jinja` (inherits the parent finetune’s chat format)
+
+> Exact files can differ by branch; see the **Files and versions** tab for each revision.
+
+---
+
+## About **Command A** (how it differs from Qwen/Qwen3 and others)
+
+- **Dense 111B** (not MoE): All parameters are active at inference; optimized for **throughput** and **enterprise reliability**.  
+- **256K context**: supports very long conversations and documents.  
+- **Enterprise agentic focus**: excels at **tool use**, **RAG**, **agents**, and **multilingual** tasks.  
+- **Efficiency**: designed for high tokens/sec and practical deployment footprints compared to similarly strong models.
+
+> See the Command A resources for details (technical report, model card, and product docs).
+
+---
+
+## Quantization recipe & implementation notes (from the attached script)
+
+The **W4A16** builds in this repo were produced with a modern **AWQ** recipe via **llm‑compressor** (AutoAWQ successor). Key choices:
+
+- **Scheme**: **W4A16**, **symmetric** INT4 weights, **group_size=128** targeting **Linear** layers.  
+- **Ignored**: `lm_head` left in higher precision.  
+- **Calibration data**: `wikitext-2-raw-v1` **train[:256]**, shuffled, preprocessed to `text`.  
+- **Calibration setup**: `num_calibration_samples=128`, `max_seq_length=256`.  
+- **Orchestration**: uses `oneshot()` to stream layers—**no manual device map / offloading**; relies on llm‑compressor’s memory management.  
+- **Export**: saved with `save_compressed=True` to include **compressed‑tensors** runtime metadata for vLLM.  
+- **Runtime dtype**: activations served in **BF16/FP16** (A16) at inference.
+
+The **INT8‑W8A16** branch follows the same structure, trading slightly higher memory for extra stability on some workloads.
+
+---
+
+## Quickstart — vLLM (compressed‑tensors)
+
+Install vLLM (recent version recommended):
+
+```bash
+pip install vllm
+```
+
+Serve (adjust to your hardware):
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve TheHouseOfTheDude/Fallen-Command-A-111B-v1_Compressed-Tensors   --quantization compressed-tensors   --tensor-parallel-size 8   --max-model-len 256000   --gpu-memory-utilization 0.70   --dtype bfloat16
+```
+
+Query via **Chat Completions**:
+
+```bash
+curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
+    "model": "TheHouseOfTheDude/Fallen-Command-A-111B-v1_Compressed-Tensors",
+    "messages": [
+      {"role":"system","content":"You are Command-A (finetuned), helpful, precise, and safe."},
+      {"role":"user","content":"Outline a retrieval pipeline for multilingual legal documents."}
+    ],
+    "max_tokens": 512,
+    "temperature": 0.7,
+    "top_p": 0.95
+  }'
+```
+
+> **Note:** `compressed‑tensors` is a **vLLM runtime format**. Loading this artifact directly in vanilla 🤗 Transformers is not supported; use vLLM for inference. For Transformers, use a different export (e.g., GPTQ/AWQ compatible) or full‑precision weights.
+
+---
+
+## Prompting / Chat Template
+
+This package follows the parent finetune’s **chat** conventions. If a `chat_template.jinja` is present in the branch, `apply_chat_template` will use it automatically.
+
+---
+
+## Lineage
+
+- **Base model:** [CohereLabs/c4ai-command-a-03-2025](https://huggingface.co/CohereLabs/c4ai-command-a-03-2025)  
+- **Finetuned parent:** [TheDrummer/Fallen-Command-A-111B-v1](https://huggingface.co/TheDrummer/Fallen-Command-A-111B-v1)  
+- **This repo:** **Quantized child** of the finetune (**compressed‑tensors** for vLLM)
+
+---
+
+## Hardware & Tips (rule‑of‑thumb)
+
+- 111B dense models typically require **multi‑GPU** deployments for best throughput.  
+- Long contexts are **KV‑cache** heavy—tune `--max-model-len` and batch size.  
+- Prefer **BF16** on GPUs with native support; otherwise **FP16**.  
+- Consider CUDA Graphs if stable in your stack.
+
+---
+
+## License & Usage
+
+This distribution inherits the licenses/policies of the **finetuned parent** and its **base** model.  
+Use of the model constitutes acceptance of the upstream terms.
+
+---
+
+## Changelog
+
+- **v1 (current)** — Quantized compressed‑tensors exports for Fallen‑Command‑A‑111B‑v1; added **W4A16** and **INT8‑W8A16** branches; model card set for **Quantized** classification.