Update README.md

README.md

@@ -1,3 +1,227 @@
----
-license: apache-2.0
----
+---
+library_name: transformers
+license: apache-2.0		
+license_link: https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model:
+- Qwen/Qwen3-235B-A22B-Instruct-2507
+---
+
+# Qwen3-235B-A22B-Instruct-2507-FP8
+<a href="https://chat.qwen.ai/" target="_blank" style="margin: 2px;">
+    <img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
+</a>
+
+## Highlights
+
+We introduce the updated version of the **Qwen3-235B-A22B-FP8 non-thinking mode**, named **Qwen3-235B-A22B-Instruct-2507-FP8**, featuring the following key enhancements:  
+
+- **Significant improvements** in general capabilities, including **instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage**.  
+- **Substantial gains** in long-tail knowledge coverage across **multiple languages**.  
+- **Markedly better alignment** with user preferences in **subjective and open-ended tasks**, enabling more helpful responses and higher-quality text generation.  
+- **Enhanced capabilities** in **256K long-context understanding**.
+
+## Model Overview
+
+This repo contains the FP8 version of **Qwen3-235B-A22B-Instruct-2507**, which has the following features:
+- Type: Causal Language Models
+- Training Stage: Pretraining & Post-training
+- Number of Parameters: 235B in total and 22B activated
+- Number of Paramaters (Non-Embedding): 234B
+- Number of Layers: 94
+- Number of Attention Heads (GQA): 64 for Q and 4 for KV
+- Number of Experts: 128
+- Number of Activated Experts: 8
+- Context Length: **262,144 natively**. 
+
+**NOTE: this model only supports the non-thinking mode. In other words, there are no ``<think></think>`` blocks in the generated outputs. Meanwhile, `enable_thinking=False` is no longer needed to be specified.**
+
+For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our [blog](https://qwenlm.github.io/blog/qwen3/), [GitHub](https://github.com/QwenLM/Qwen3), and [Documentation](https://qwen.readthedocs.io/en/latest/).
+
+
+## Performance
+
+
+|  | Deepseek V3-0324 | GPT-4o-0327 | Claude Opus 4 non-thinking | Kimi-K2 | Qwen3-235B-A22B non-thinking | Qwen3-235B-A22B-Instruct-2507 |
+|--- | --- | --- | --- | --- | --- | ---|
+| **Knowledge** | | | | | | |
+| MMLU | 88.3 | 89.7 | 91.7 | 89.5 | 86.0 | 89.7 |
+| MMLU-Pro | 81.2 | 79.8 | 83.7 | 81.1 | 75.2 | 83.0 |
+| MMLU-Redux | 90.4 | 91.3 | 93.1 | 92.7 | 89.2 | 93.1 |
+| GPQA | 68.4 | 66.9 | 88.9 | 75.1 | 62.9 | 77.5 |
+| SuperGPQA | 57.3 | 51.0 | 58.0 | 57.2 | 48.2 | 62.6 |
+| SimpleQA | 27.2 | 40.3 | 15.9 | 31.0 | 12.2 | 54.3 |
+| CSimpleQA | 71.1 | 60.2 | 59.5 | 74.5 | 60.8 | 84.3 |
+| **Reasoning** | | | | | | |
+| AIME24 | 59.4 | 32.5 | 43.4 | 69.6 | 40.1 | 82.0 |
+| AIME25 | 46.6 | 26.7 | 33.1 | 49.5 | 24.7 | 70.3 |
+| HMMT25 | 25.0 | 7.92 | 15.42 | ？ | 10.0 | 55.4 |
+| ArcAGI | 9.0 | 8.8 | 28.3 | 13.3 | 4.3 | 41.8 |
+| ZebraLogic | 83.4 | 52.6 | 79.7 | 89.0 | 37.7 | 95.0 |
+| LiveBench1125 | 66.8 | 63.7 | 74.8 | 76.4 | 62.5 | 75.4 |
+| **Coding** | | | | | | |
+| LCBv6 (25.02 - 25.05) | 45.2 | 35.8 | 44.6 | 48.9 | ing | 51.8 |
+| MultiPL-E | 82.2 | 82.7 | 88.5 | 83.1 | ing | 87.9 |
+| Aider | 55.1 | 45.3 | 70.7 | 59.0 | 59.6 | 57.3 |
+| **Instruction Following** | | | | | | |
+| SIFO | 62.5 | 64.9 | 75.8 | 60.6 | 53.2 | 58.5 |
+| SIFO-multiturn | 59.1 | 64.4 | 66.9 | 62.7 | 47.3 | 61.9 |
+| IFEval | 82.3 | 83.9 | 88.9 | 89.8 | 83.2 | 88.7 |
+| **Open-ended tasks** | | | | | | |
+| Arena-Hard v2 (win rate) gpt-4.1 as judger | 45.6 | 61.9 | 46.6 | 66.1 | 52.0 | 79.2 |
+| Creative Writing v3 | 81.6 | 84.9 | 83.1 | 88.1 | 80.4 | 87.5 |
+| WritingBench | 74.5 | 75.5 | 79.7 | 86.2 | 77.0 | 85.2 |
+| **Agent** | | | | | | |
+| BFCL-v3 | 64.7 | 66.5 | 60.1 | 65.2 | 68.0 | 70.9 |
+| TAU-Retail | 49.6 | 60.3（gpt-4o-20241120） | 81.4* | 70.7 | 65.2 | 71.3 |
+| TAU-Airline | 32.0 | 42.8（gpt-4o-20241120） | 59.6* | 53.5 | 32.0 | 44.0 |
+| **Multilingualism** | | | | | | |
+| MultiIF | 66.5 | 70.4 | - | 76.2 | 70.2 | 77.5 |
+| MMLU-ProX | 75.8 | 76.2 | - | 74.5 | 73.2 | 79.4 |
+| INCLUDE | 80.1 | 82.1 | - | 76.9 | 75.6 | 79.5 |
+| PolyMATH | 32.2 | 25.5 | 30.0 | 44.8 | 27.0 | 50.2 |
+
+
+
+## Quickstart
+
+The code of Qwen3-MoE has been in the latest Hugging Face `transformers` and we advise you to use the latest version of `transformers`.
+
+With `transformers<4.51.0`, you will encounter the following error:
+```
+KeyError: 'qwen3_moe'
+```
+
+The following contains a code snippet illustrating how to use the model generate content based on given inputs. 
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "Qwen/Qwen3-235B-A22B-Instruct-2507-FP8"
+
+# load the tokenizer and the model
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto"
+)
+
+# prepare the model input
+prompt = "Give me a short introduction to large language model."
+messages = [
+    {"role": "user", "content": prompt}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+# conduct text completion
+generated_ids = model.generate(
+    **model_inputs,
+    max_new_tokens=16384
+)
+output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 
+
+content = tokenizer.decode(output_ids, skip_special_tokens=True)
+
+print("content:", content)
+```
+
+For deployment, you can use `sglang>=0.4.6.post1` or `vllm>=0.8.5` or to create an OpenAI-compatible API endpoint:
+- SGLang:
+    ```shell
+    python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --tp 4 --context-length 262144
+    ```
+- vLLM:
+    ```shell
+    vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --tensor-parallel-size 4 --max-model-len 262144
+    ```
+
+Note: If you encounter out-of-memory (OOM) issues, consider reducing the context length to a shorter value, such as 32,768.
+
+For local use, applications such as Ollama, LMStudio, MLX-LM, llama.cpp, and KTransformers have also supported Qwen3.
+
+## Note on FP8
+
+For convenience and performance, we have provided `fp8`-quantized model checkpoint for Qwen3, whose name ends with `-FP8`. The quantization method is fine-grained `fp8` quantization with block size of 128. You can find more details in the `quantization_config` field in `config.json`.
+
+You can use the Qwen3-235B-A22B-Instruct-2507-FP8 model with serveral inference frameworks, including `transformers`, `sglang`, and `vllm`, as the original bfloat16 model.
+However, please pay attention to the following known issues:
+- `transformers`:
+    - there are currently issues with the "fine-grained fp8" method in `transformers` for distributed inference. You may need to set the environment variable `CUDA_LAUNCH_BLOCKING=1` if multiple devices are used in inference.
+
+## Agentic Use
+
+Qwen3 excels in tool calling capabilities. We recommend using [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) to make the best use of agentic ability of Qwen3. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity.
+
+To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself.
+```python
+from qwen_agent.agents import Assistant
+
+# Define LLM
+llm_cfg = {
+    'model': 'Qwen3-235B-A22B-Instruct-2507-FP8',
+
+    # Use a custom endpoint compatible with OpenAI API:
+    'model_server': 'http://localhost:8000/v1',  # api_base
+    'api_key': 'EMPTY',
+}
+
+# Define Tools
+tools = [
+    {'mcpServers': {  # You can specify the MCP configuration file
+            'time': {
+                'command': 'uvx',
+                'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']
+            },
+            "fetch": {
+                "command": "uvx",
+                "args": ["mcp-server-fetch"]
+            }
+        }
+    },
+  'code_interpreter',  # Built-in tools
+]
+
+# Define Agent
+bot = Assistant(llm=llm_cfg, function_list=tools)
+
+# Streaming generation
+messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]
+for responses in bot.run(messages=messages):
+    pass
+print(responses)
+```
+
+## Best Practices
+
+To achieve optimal performance, we recommend the following settings:
+
+1. **Sampling Parameters**:
+   - We suggest using `Temperature=0.7`, `TopP=0.8`, `TopK=20`, and `MinP=0`.
+   - For supported frameworks, you can adjust the `presence_penalty` parameter between 0 and 2 to reduce endless repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
+
+2. **Adequate Output Length**: We recommend using an output length of 16,384 tokens for most queries, which is adequate for instruct models.
+
+3. **Standardize Output Format**: We recommend using prompts to standardize model outputs when benchmarking.
+   - **Math Problems**: Include "Please reason step by step, and put your final answer within \boxed{}." in the prompt.
+   - **Multiple-Choice Questions**: Add the following JSON structure to the prompt to standardize responses: "Please show your choice in the `answer` field with only the choice letter, e.g., `"answer": "C"`."
+
+### Citation
+
+If you find our work helpful, feel free to give us a cite.
+
+```
+@misc{qwen3technicalreport,
+      title={Qwen3 Technical Report}, 
+      author={Qwen Team},
+      year={2025},
+      eprint={2505.09388},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2505.09388}, 
+}
+```