Update README.md
Browse files
README.md
CHANGED
|
@@ -1,788 +1,12 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
---
|
| 6 |
-
<div align="center">
|
| 7 |
-
<picture>
|
| 8 |
-
<img src="figures/kimi-logo.png" width="30%" alt="Kimi K2: Open Agentic Intellignece">
|
| 9 |
-
</picture>
|
| 10 |
-
</div>
|
| 11 |
|
| 12 |
-
<
|
| 13 |
|
| 14 |
-
|
| 15 |
-
<a href="https://www.kimi.com" target="_blank"><img alt="Chat" src="https://img.shields.io/badge/🤖%20Chat-Kimi%20K2-ff6b6b?color=1783ff&logoColor=white"/></a>
|
| 16 |
-
<a href="https://www.moonshot.ai" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Moonshot%20AI-white?logo=Kimi&logoColor=white"/></a>
|
| 17 |
-
</div>
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
<a href="https://twitter.com/kimi_moonshot" target="_blank"><img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-Kimi.ai-white?logo=x&logoColor=white"/></a>
|
| 22 |
-
<a href="https://discord.gg/TYU2fdJykW" target="_blank"><img alt="Discord" src="https://img.shields.io/badge/Discord-Kimi.ai-white?logo=discord&logoColor=white"/></a>
|
| 23 |
-
</div>
|
| 24 |
-
|
| 25 |
-
<div align="center" style="line-height: 1;">
|
| 26 |
-
<a href="https://github.com/moonshotai/Kimi-K2/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Modified_MIT-f5de53?&color=f5de53"/></a>
|
| 27 |
-
</div>
|
| 28 |
-
|
| 29 |
-
<p align="center">
|
| 30 |
-
<b>📰 <a href="https://moonshotai.github.io/Kimi-K2/">Tech Blog</a></b> | <b>📄 Paper Link (comming soon)</b>
|
| 31 |
-
</p>
|
| 32 |
-
|
| 33 |
-
## 1. Model Introduction
|
| 34 |
-
|
| 35 |
-
Kimi K2 is a state-of-the-art mixture-of-experts (MoE) language model with 32 billion activated parameters and 1 trillion total parameters. Trained with the Muon optimizer, Kimi K2 achieves exceptional performance across frontier knowledge, reasoning, and coding tasks while being meticulously optimized for agentic capabilities.
|
| 36 |
-
|
| 37 |
-
### Key Features
|
| 38 |
-
- Large-Scale Training: Pre-trained a 1T parameter MoE model on 15.5T tokens with zero training instability.
|
| 39 |
-
- MuonClip Optimizer: We apply the Muon optimizer to an unprecedented scale, and develop novel optimization techniques to resolve instabilities while scaling up.
|
| 40 |
-
- Agentic Intelligence: Specifically designed for tool use, reasoning, and autonomous problem-solving.
|
| 41 |
-
|
| 42 |
-
### Model Variants
|
| 43 |
-
- **Kimi-K2-Base**: The foundation model, a strong start for researchers and builders who want full control for fine-tuning and custom solutions.
|
| 44 |
-
- **Kimi-K2-Instruct**: The post-trained model best for drop-in, general-purpose chat and agentic experiences. It is a reflex-grade model without long thinking.
|
| 45 |
-
|
| 46 |
-
<div align="center">
|
| 47 |
-
<picture>
|
| 48 |
-
<img src="figures/banner.png" width="80%" alt="Evaluation Results">
|
| 49 |
-
</picture>
|
| 50 |
-
</div>
|
| 51 |
-
|
| 52 |
-
## 2. Model Summary
|
| 53 |
-
|
| 54 |
-
<div align="center">
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
| | |
|
| 58 |
-
|:---:|:---:|
|
| 59 |
-
| **Architecture** | Mixture-of-Experts (MoE) |
|
| 60 |
-
| **Total Parameters** | 1T |
|
| 61 |
-
| **Activated Parameters** | 32B |
|
| 62 |
-
| **Number of Layers** (Dense layer included) | 61 |
|
| 63 |
-
| **Number of Dense Layers** | 1 |
|
| 64 |
-
| **Attention Hidden Dimension** | 7168 |
|
| 65 |
-
| **MoE Hidden Dimension** (per Expert) | 2048 |
|
| 66 |
-
| **Number of Attention Heads** | 64 |
|
| 67 |
-
| **Number of Experts** | 384 |
|
| 68 |
-
| **Selected Experts per Token** | 8 |
|
| 69 |
-
| **Number of Shared Experts** | 1 |
|
| 70 |
-
| **Vocabulary Size** | 160K |
|
| 71 |
-
| **Context Length** | 128K |
|
| 72 |
-
| **Attention Mechanism** | MLA |
|
| 73 |
-
| **Activation Function** | SwiGLU |
|
| 74 |
-
</div>
|
| 75 |
-
|
| 76 |
-
## 3. Evaluation Results
|
| 77 |
-
|
| 78 |
-
#### Instruction model evaluation results
|
| 79 |
-
|
| 80 |
-
<div align="center">
|
| 81 |
-
<table>
|
| 82 |
-
<thead>
|
| 83 |
-
<tr>
|
| 84 |
-
<th align="center">Benchmark</th>
|
| 85 |
-
<th align="center">Metric</th>
|
| 86 |
-
<th align="center"><sup>Kimi K2 Instruct</sup></th>
|
| 87 |
-
<th align="center"><sup>DeepSeek-V3-0324</sup></th>
|
| 88 |
-
<th align="center"><sup>Qwen3-235B-A22B <br><sup>(non-thinking)</sup></sup></th>
|
| 89 |
-
<th align="center"><sup>Claude Sonnet 4 <br><sup>(w/o extended thinking)</sup></sup></th>
|
| 90 |
-
<th align="center"><sup>Claude Opus 4 <br><sup>(w/o extended thinking)</sup></sup></th>
|
| 91 |
-
<th align="center"><sup>GPT-4.1</sup></th>
|
| 92 |
-
<th align="center"><sup>Gemini 2.5 Flash <br> Preview (05-20)</sup></th>
|
| 93 |
-
</tr>
|
| 94 |
-
</thead>
|
| 95 |
-
<tbody>
|
| 96 |
-
<tr>
|
| 97 |
-
<td align="center" colspan=9><strong>Coding Tasks</strong></td>
|
| 98 |
-
</tr>
|
| 99 |
-
<tr>
|
| 100 |
-
<td align="center">LiveCodeBench v6<br><sup>(Aug 24 - May 25)</sup></td>
|
| 101 |
-
<td align="center">Pass@1</td>
|
| 102 |
-
<td align="center"><strong>53.7</strong></td>
|
| 103 |
-
<td align="center">46.9</td>
|
| 104 |
-
<td align="center">37.0</td>
|
| 105 |
-
<td align="center">48.5</td>
|
| 106 |
-
<td align="center">47.4</td>
|
| 107 |
-
<td align="center">44.7</td>
|
| 108 |
-
<td align="center">44.7</td>
|
| 109 |
-
</tr>
|
| 110 |
-
<tr>
|
| 111 |
-
<td align="center">OJBench</td>
|
| 112 |
-
<td align="center">Pass@1</td>
|
| 113 |
-
<td align="center"><strong>27.1</strong></td>
|
| 114 |
-
<td align="center">24.0</td>
|
| 115 |
-
<td align="center">11.3</td>
|
| 116 |
-
<td align="center">15.3</td>
|
| 117 |
-
<td align="center">19.6</td>
|
| 118 |
-
<td align="center">19.5</td>
|
| 119 |
-
<td align="center">19.5</td>
|
| 120 |
-
</tr>
|
| 121 |
-
|
| 122 |
-
<tr>
|
| 123 |
-
<td align="center">MultiPL-E</td>
|
| 124 |
-
<td align="center">Pass@1</td>
|
| 125 |
-
<td align="center"><ins><strong>85.7</strong></ins></td>
|
| 126 |
-
<td align="center">83.1</td>
|
| 127 |
-
<td align="center">78.2</td>
|
| 128 |
-
<td align="center">88.6</td>
|
| 129 |
-
<td align="center"><strong>89.6</strong></td>
|
| 130 |
-
<td align="center">86.7</td>
|
| 131 |
-
<td align="center">85.6</td>
|
| 132 |
-
</tr>
|
| 133 |
-
|
| 134 |
-
<tr>
|
| 135 |
-
<td align="center">SWE-bench Verified <br/><sup>(Agentless Coding)</sup></td>
|
| 136 |
-
<td align="center">Single Patch w/o Test (Acc)</td>
|
| 137 |
-
<td align="center"><ins><strong>51.8</strong></ins></td>
|
| 138 |
-
<td align="center">36.6</td>
|
| 139 |
-
<td align="center">39.4</td>
|
| 140 |
-
<td align="center">50.2</td>
|
| 141 |
-
<td align="center"><strong>53.0</strong></td>
|
| 142 |
-
<td align="center">40.8</td>
|
| 143 |
-
<td align="center">32.6</td>
|
| 144 |
-
</tr>
|
| 145 |
-
|
| 146 |
-
<tr>
|
| 147 |
-
<td align="center" rowspan="2">SWE-bench Verified <br/> <sup>(Agentic Coding)</sup></td>
|
| 148 |
-
<td align="center">Single Attempt (Acc)</td>
|
| 149 |
-
<td align="center"><ins><strong>65.8</strong></ins></td>
|
| 150 |
-
<td align="center">38.8</td>
|
| 151 |
-
<td align="center">34.4</td>
|
| 152 |
-
<td align="center"><strong>72.7</strong><sup>*</sup></td>
|
| 153 |
-
<td align="center">72.5<sup>*</sup></td>
|
| 154 |
-
<td align="center">54.6</td>
|
| 155 |
-
<td align="center">—</td>
|
| 156 |
-
</tr>
|
| 157 |
-
|
| 158 |
-
<tr>
|
| 159 |
-
<!--<td align="center">(Agentic Coding)</td>-->
|
| 160 |
-
<td align="center">Multiple Attempts (Acc)</td>
|
| 161 |
-
<td align="center"><ins><strong>71.6</strong></ins></td>
|
| 162 |
-
<td align="center">—</td>
|
| 163 |
-
<td align="center">—</td>
|
| 164 |
-
<td align="center"><strong>80.2</strong></td>
|
| 165 |
-
<td align="center">79.4<sup>*</sup></td>
|
| 166 |
-
<td align="center">—</td>
|
| 167 |
-
<td align="center">—</td>
|
| 168 |
-
</tr>
|
| 169 |
-
|
| 170 |
-
<tr>
|
| 171 |
-
<td align="center">SWE-bench Multilingual<br /> <sup>(Agentic Coding)</sup></td>
|
| 172 |
-
<td align="center">Single Attempt (Acc)</td>
|
| 173 |
-
<td align="center"><ins><strong>47.3</strong> </ins></td>
|
| 174 |
-
<td align="center">25.8</td>
|
| 175 |
-
<td align="center">20.9</td>
|
| 176 |
-
<td align="center"><strong>51.0</strong></td>
|
| 177 |
-
<td align="center">—</td>
|
| 178 |
-
<td align="center">31.5</td>
|
| 179 |
-
<td align="center">—</td>
|
| 180 |
-
</tr>
|
| 181 |
-
|
| 182 |
-
<tr>
|
| 183 |
-
<td align="center" rowspan="2">TerminalBench</td>
|
| 184 |
-
<td align="center">Inhouse Framework (Acc)</td>
|
| 185 |
-
<td align="center"><ins><strong>30.0</strong></ins></td>
|
| 186 |
-
<td align="center">—</td>
|
| 187 |
-
<td align="center">—</td>
|
| 188 |
-
<td align="center">35.5</td>
|
| 189 |
-
<td align="center"><strong>43.2</strong></td>
|
| 190 |
-
<td align="center">8.3</td>
|
| 191 |
-
<td align="center">—</td>
|
| 192 |
-
</tr>
|
| 193 |
-
|
| 194 |
-
<tr>
|
| 195 |
-
<!--<td align="center">TerminalBench</td>-->
|
| 196 |
-
<td align="center">Terminus (Acc)</td>
|
| 197 |
-
<td align="center"><ins><strong>25.0</strong> </ins></td>
|
| 198 |
-
<td align="center">16.3</td>
|
| 199 |
-
<td align="center">6.6</td>
|
| 200 |
-
<td align="center">—</td>
|
| 201 |
-
<td align="center">—</td>
|
| 202 |
-
<td align="center"><strong>30.3</strong></td>
|
| 203 |
-
<td align="center">16.8</td>
|
| 204 |
-
</tr>
|
| 205 |
-
<tr>
|
| 206 |
-
<td align="center">Aider-Polyglot</td>
|
| 207 |
-
<td align="center">Acc</td>
|
| 208 |
-
<td align="center">60.0</td>
|
| 209 |
-
<td align="center">55.1</td>
|
| 210 |
-
<td align="center"><ins><strong>61.8</strong></ins></td>
|
| 211 |
-
<td align="center">56.4</td>
|
| 212 |
-
<td align="center"><strong>70.7</strong></td>
|
| 213 |
-
<td align="center">52.4</td>
|
| 214 |
-
<td align="center">44.0</td>
|
| 215 |
-
</tr>
|
| 216 |
-
<tr>
|
| 217 |
-
<td align="center" colspan=9><strong>Tool Use Tasks</strong></td>
|
| 218 |
-
</tr>
|
| 219 |
-
<tr>
|
| 220 |
-
<td align="center">Tau2 retail</td>
|
| 221 |
-
<td align="center">Avg@4</td>
|
| 222 |
-
<td align="center"><ins><strong>70.6</strong></ins></td>
|
| 223 |
-
<td align="center">69.1</td>
|
| 224 |
-
<td align="center">57.0</td>
|
| 225 |
-
<td align="center">75.0</td>
|
| 226 |
-
<td align="center"><strong>81.8</strong></td>
|
| 227 |
-
<td align="center">74.8</td>
|
| 228 |
-
<td align="center">64.3</td>
|
| 229 |
-
</tr>
|
| 230 |
-
<tr>
|
| 231 |
-
<td align="center">Tau2 airline</td>
|
| 232 |
-
<td align="center">Avg@4</td>
|
| 233 |
-
<td align="center"><ins><strong>56.5</strong></ins></td>
|
| 234 |
-
<td align="center">39.0</td>
|
| 235 |
-
<td align="center">26.5</td>
|
| 236 |
-
<td align="center">55.5</td>
|
| 237 |
-
<td align="center"><strong>60.0</strong></td>
|
| 238 |
-
<td align="center">54.5</td>
|
| 239 |
-
<td align="center">42.5</td>
|
| 240 |
-
</tr>
|
| 241 |
-
<tr>
|
| 242 |
-
<td align="center">Tau2 telecom</td>
|
| 243 |
-
<td align="center">Avg@4</td>
|
| 244 |
-
<td align="center"><strong>65.8</strong></td>
|
| 245 |
-
<td align="center">32.5</td>
|
| 246 |
-
<td align="center">22.1</td>
|
| 247 |
-
<td align="center">45.2</td>
|
| 248 |
-
<td align="center">57.0</td>
|
| 249 |
-
<td align="center">38.6</td>
|
| 250 |
-
<td align="center">16.9</td>
|
| 251 |
-
</tr>
|
| 252 |
-
<tr>
|
| 253 |
-
<td align="center">AceBench</td>
|
| 254 |
-
<td align="center">Acc</td>
|
| 255 |
-
<td align="center"><ins><strong>76.5</strong></ins></td>
|
| 256 |
-
<td align="center">72.7</td>
|
| 257 |
-
<td align="center">70.5</td>
|
| 258 |
-
<td align="center">76.2</td>
|
| 259 |
-
<td align="center">75.6</td>
|
| 260 |
-
<td align="center"><strong>80.1</strong></td>
|
| 261 |
-
<td align="center">74.5</td>
|
| 262 |
-
</tr>
|
| 263 |
-
<tr>
|
| 264 |
-
<td align="center" colspan=9><strong>Math & STEM Tasks</strong></td>
|
| 265 |
-
</tr>
|
| 266 |
-
<tr>
|
| 267 |
-
<td align="center">AIME 2024</td>
|
| 268 |
-
<td align="center">Avg@64</td>
|
| 269 |
-
<td align="center"><strong>69.6</strong></td>
|
| 270 |
-
<td align="center">59.4<sup>*</sup></td>
|
| 271 |
-
<td align="center">40.1<sup>*</sup></td>
|
| 272 |
-
<td align="center">43.4</td>
|
| 273 |
-
<td align="center">48.2</td>
|
| 274 |
-
<td align="center">46.5</td>
|
| 275 |
-
<td align="center">61.3</td>
|
| 276 |
-
</tr>
|
| 277 |
-
<tr>
|
| 278 |
-
<td align="center">AIME 2025</td>
|
| 279 |
-
<td align="center">Avg@64</td>
|
| 280 |
-
<td align="center"><strong>49.5</strong></td>
|
| 281 |
-
<td align="center">46.7</td>
|
| 282 |
-
<td align="center">24.7<sup>*</sup></td>
|
| 283 |
-
<td align="center">33.1<sup>*</sup></td>
|
| 284 |
-
<td align="center">33.9<sup>*</sup></td>
|
| 285 |
-
<td align="center">37.0</td>
|
| 286 |
-
<td align="center">46.6</td>
|
| 287 |
-
</tr>
|
| 288 |
-
<tr>
|
| 289 |
-
<td align="center">MATH-500</td>
|
| 290 |
-
<td align="center">Acc</td>
|
| 291 |
-
<td align="center"><strong>97.4</strong></td>
|
| 292 |
-
<td align="center">94.0<sup>*</sup></td>
|
| 293 |
-
<td align="center">91.2<sup>*</sup></td>
|
| 294 |
-
<td align="center">94.0</td>
|
| 295 |
-
<td align="center">94.4</td>
|
| 296 |
-
<td align="center">92.4</td>
|
| 297 |
-
<td align="center">95.4</td>
|
| 298 |
-
</tr>
|
| 299 |
-
<tr>
|
| 300 |
-
<td align="center">HMMT 2025</td>
|
| 301 |
-
<td align="center">Avg@32</td>
|
| 302 |
-
<td align="center"><strong>38.8</strong></td>
|
| 303 |
-
<td align="center">27.5</td>
|
| 304 |
-
<td align="center">11.9</td>
|
| 305 |
-
<td align="center">15.9</td>
|
| 306 |
-
<td align="center">15.9</td>
|
| 307 |
-
<td align="center">19.4</td>
|
| 308 |
-
<td align="center">34.7</td>
|
| 309 |
-
</tr>
|
| 310 |
-
<tr>
|
| 311 |
-
<td align="center">CNMO 2024</td>
|
| 312 |
-
<td align="center">Avg@16</td>
|
| 313 |
-
<td align="center">74.3</td>
|
| 314 |
-
<td align="center"><ins><strong>74.7</strong></ins></td>
|
| 315 |
-
<td align="center">48.6</td>
|
| 316 |
-
<td align="center">60.4</td>
|
| 317 |
-
<td align="center">57.6</td>
|
| 318 |
-
<td align="center">56.6</td>
|
| 319 |
-
<td align="center"><strong>75.0</strong></td>
|
| 320 |
-
</tr>
|
| 321 |
-
<tr>
|
| 322 |
-
<td align="center">PolyMath-en</td>
|
| 323 |
-
<td align="center">Avg@4</td>
|
| 324 |
-
<td align="center"><strong>65.1</strong></td>
|
| 325 |
-
<td align="center">59.5</td>
|
| 326 |
-
<td align="center">51.9</td>
|
| 327 |
-
<td align="center">52.8</td>
|
| 328 |
-
<td align="center">49.8</td>
|
| 329 |
-
<td align="center">54.0</td>
|
| 330 |
-
<td align="center">49.9</td>
|
| 331 |
-
</tr>
|
| 332 |
-
|
| 333 |
-
<tr>
|
| 334 |
-
<td align="center">ZebraLogic</td>
|
| 335 |
-
<td align="center">Acc</td>
|
| 336 |
-
<td align="center"><strong>89.0</strong></td>
|
| 337 |
-
<td align="center">84.0</td>
|
| 338 |
-
<td align="center">37.7<sup>*</sup></td>
|
| 339 |
-
<td align="center">73.7</td>
|
| 340 |
-
<td align="center">59.3</td>
|
| 341 |
-
<td align="center">58.5</td>
|
| 342 |
-
<td align="center">57.9</td>
|
| 343 |
-
</tr>
|
| 344 |
-
|
| 345 |
-
<tr>
|
| 346 |
-
<td align="center">AutoLogi</td>
|
| 347 |
-
<td align="center">Acc</td>
|
| 348 |
-
<td align="center"><ins><strong>89.5</strong></ins></td>
|
| 349 |
-
<td align="center">88.9</td>
|
| 350 |
-
<td align="center">83.3</td>
|
| 351 |
-
<td align="center"><strong>89.8</strong></td>
|
| 352 |
-
<td align="center">86.1</td>
|
| 353 |
-
<td align="center">88.2</td>
|
| 354 |
-
<td align="center">84.1</td>
|
| 355 |
-
</tr>
|
| 356 |
-
|
| 357 |
-
<tr>
|
| 358 |
-
<td align="center">GPQA-Diamond</td>
|
| 359 |
-
<td align="center">Avg@8</td>
|
| 360 |
-
<td align="center"><strong>75.1</strong></td>
|
| 361 |
-
<td align="center">68.4<sup>*</sup></td>
|
| 362 |
-
<td align="center">62.9<sup>*</sup></td>
|
| 363 |
-
<td align="center">70.0<sup>*</sup></td>
|
| 364 |
-
<td align="center">74.9<sup>*</sup></td>
|
| 365 |
-
<td align="center">66.3</td>
|
| 366 |
-
<td align="center">68.2</td>
|
| 367 |
-
</tr>
|
| 368 |
-
|
| 369 |
-
<tr>
|
| 370 |
-
<td align="center">SuperGPQA</td>
|
| 371 |
-
<td align="center">Acc</td>
|
| 372 |
-
<td align="center"><strong>57.2</strong></td>
|
| 373 |
-
<td align="center">53.7</td>
|
| 374 |
-
<td align="center">50.2</td>
|
| 375 |
-
<td align="center">55.7</td>
|
| 376 |
-
<td align="center">56.5</td>
|
| 377 |
-
<td align="center">50.8</td>
|
| 378 |
-
<td align="center">49.6</td>
|
| 379 |
-
</tr>
|
| 380 |
-
|
| 381 |
-
<tr>
|
| 382 |
-
<td align="center">Humanity's Last Exam<br><sup>(Text Only)</sup></td>
|
| 383 |
-
<td align="center">-</td>
|
| 384 |
-
<td align="center">4.7</td>
|
| 385 |
-
<td align="center">5.2</td>
|
| 386 |
-
<td align="center"><ins><strong>5.7</strong></ins></td>
|
| 387 |
-
<td align="center">5.8</td>
|
| 388 |
-
<td align="center"><strong>7.1</strong></td>
|
| 389 |
-
<td align="center">3.7</td>
|
| 390 |
-
<td align="center">5.6</td>
|
| 391 |
-
</tr>
|
| 392 |
-
|
| 393 |
-
<tr>
|
| 394 |
-
<td align="center" colspan=9><strong>General Tasks</strong></td>
|
| 395 |
-
</tr>
|
| 396 |
-
|
| 397 |
-
<tr>
|
| 398 |
-
<td align="center">MMLU</td>
|
| 399 |
-
<td align="center">EM</td>
|
| 400 |
-
<td align="center"><ins><strong>89.5</strong></ins></td>
|
| 401 |
-
<td align="center">89.4</td>
|
| 402 |
-
<td align="center">87.0</td>
|
| 403 |
-
<td align="center">91.5</td>
|
| 404 |
-
<td align="center"><strong>92.9</strong></td>
|
| 405 |
-
<td align="center">90.4</td>
|
| 406 |
-
<td align="center">90.1</td>
|
| 407 |
-
</tr>
|
| 408 |
-
|
| 409 |
-
<tr>
|
| 410 |
-
<td align="center">MMLU-Redux</td>
|
| 411 |
-
<td align="center">EM</td>
|
| 412 |
-
<td align="center"><ins><strong>92.7</strong></ins></td>
|
| 413 |
-
<td align="center">90.5</td>
|
| 414 |
-
<td align="center">89.2</td>
|
| 415 |
-
<td align="center">93.6</td>
|
| 416 |
-
<td align="center"><strong>94.2</strong></td>
|
| 417 |
-
<td align="center">92.4</td>
|
| 418 |
-
<td align="center">90.6</td>
|
| 419 |
-
</tr>
|
| 420 |
-
|
| 421 |
-
<tr>
|
| 422 |
-
<td align="center">MMLU-Pro</td>
|
| 423 |
-
<td align="center">EM</td>
|
| 424 |
-
<td align="center">81.1</td>
|
| 425 |
-
<td align="center"><ins><strong>81.2</strong></ins><sup>*</sup></td>
|
| 426 |
-
<td align="center">77.3</td>
|
| 427 |
-
<td align="center">83.7</td>
|
| 428 |
-
<td align="center"><strong>86.6</strong></td>
|
| 429 |
-
<td align="center">81.8</td>
|
| 430 |
-
<td align="center">79.4</td>
|
| 431 |
-
</tr>
|
| 432 |
-
|
| 433 |
-
<tr>
|
| 434 |
-
<td align="center">IFEval</td>
|
| 435 |
-
<td align="center">Prompt Strict</td>
|
| 436 |
-
<td align="center"><strong>89.8</strong></td>
|
| 437 |
-
<td align="center">81.1</td>
|
| 438 |
-
<td align="center">83.2<sup>*</sup></td>
|
| 439 |
-
<td align="center">87.6</td>
|
| 440 |
-
<td align="center">87.4</td>
|
| 441 |
-
<td align="center">88.0</td>
|
| 442 |
-
<td align="center">84.3</td>
|
| 443 |
-
</tr>
|
| 444 |
-
|
| 445 |
-
<tr>
|
| 446 |
-
<td align="center">Multi-Challenge</td>
|
| 447 |
-
<td align="center">Acc</td>
|
| 448 |
-
<td align="center"><strong>54.1</strong></td>
|
| 449 |
-
<td align="center">31.4</td>
|
| 450 |
-
<td align="center">34.0</td>
|
| 451 |
-
<td align="center">46.8</td>
|
| 452 |
-
<td align="center">49.0</td>
|
| 453 |
-
<td align="center">36.4</td>
|
| 454 |
-
<td align="center">39.5</td>
|
| 455 |
-
</tr>
|
| 456 |
-
|
| 457 |
-
<tr>
|
| 458 |
-
<td align="center">SimpleQA</td>
|
| 459 |
-
<td align="center">Correct</td>
|
| 460 |
-
<td align="center"><ins><strong>31.0</strong></ins></td>
|
| 461 |
-
<td align="center">27.7</td>
|
| 462 |
-
<td align="center">13.2</td>
|
| 463 |
-
<td align="center">15.9</td>
|
| 464 |
-
<td align="center">22.8</td>
|
| 465 |
-
<td align="center"><strong>42.3</strong></td>
|
| 466 |
-
<td align="center">23.3</td>
|
| 467 |
-
</tr>
|
| 468 |
-
|
| 469 |
-
<tr>
|
| 470 |
-
<td align="center">Livebench</td>
|
| 471 |
-
<td align="center">Pass@1</td>
|
| 472 |
-
<td align="center"><strong>76.4</strong></td>
|
| 473 |
-
<td align="center">72.4</td>
|
| 474 |
-
<td align="center">67.6</td>
|
| 475 |
-
<td align="center">74.8</td>
|
| 476 |
-
<td align="center">74.6</td>
|
| 477 |
-
<td align="center">69.8</td>
|
| 478 |
-
<td align="center">67.8</td>
|
| 479 |
-
</tr>
|
| 480 |
-
</tbody>
|
| 481 |
-
</table>
|
| 482 |
-
</div>
|
| 483 |
-
<sup>
|
| 484 |
-
• Bold denotes global SOTA, and underlined denotes open-source SOTA.
|
| 485 |
-
</sup><br/><sup>
|
| 486 |
-
• Data points marked with * are taken directly from the model's tech report or blog.
|
| 487 |
-
</sup><br/><sup>
|
| 488 |
-
• All metrics, except for SWE-bench Verified (Agentless), are evaluated with an 8k output token length. SWE-bench Verified (Agentless) is limited to a 16k output token length.
|
| 489 |
-
</sup><br/><sup>
|
| 490 |
-
• Kimi K2 achieves 65.8% pass@1 on the SWE-bench Verified tests with bash/editor tools (single-attempt patches, no test-time compute). It also achieves a 47.3% pass@1 on the SWE-bench Multilingual tests under the same conditions. Additionally, we report results on SWE-bench Verified tests (71.6%) that leverage parallel test-time compute by sampling multiple sequences and selecting the single best via an internal scoring model.
|
| 491 |
-
</sup><br/><sup>
|
| 492 |
-
• To ensure the stability of the evaluation, we employed avg@k on the AIME, HMMT, CNMO, PolyMath-en, GPQA-Diamond, EvalPlus, Tau2.
|
| 493 |
-
</sup><br/><sup>
|
| 494 |
-
• Some data points have been omitted due to prohibitively expensive evaluation costs.
|
| 495 |
-
</sup>
|
| 496 |
-
|
| 497 |
-
---
|
| 498 |
-
|
| 499 |
-
#### Base model evaluation results
|
| 500 |
-
|
| 501 |
-
<div align="center">
|
| 502 |
-
|
| 503 |
-
<table>
|
| 504 |
-
<thead>
|
| 505 |
-
<tr>
|
| 506 |
-
<th align="center">Benchmark</th>
|
| 507 |
-
<th align="center">Metric</th>
|
| 508 |
-
<th align="center">Shot</th>
|
| 509 |
-
<th align="center">Kimi K2 Base</th>
|
| 510 |
-
<th align="center">Deepseek-V3-Base</th>
|
| 511 |
-
<th align="center">Qwen2.5-72B</th>
|
| 512 |
-
<th align="center">Llama 4 Maverick</th>
|
| 513 |
-
</tr>
|
| 514 |
-
</thead>
|
| 515 |
-
<tbody>
|
| 516 |
-
<tr>
|
| 517 |
-
<td align="center" colspan="7"><strong>General Tasks</strong></td>
|
| 518 |
-
</tr>
|
| 519 |
-
<tr>
|
| 520 |
-
<td align="center">MMLU</td>
|
| 521 |
-
<td align="center">EM</td>
|
| 522 |
-
<td align="center">5-shot</td>
|
| 523 |
-
<td align="center"><strong>87.8</strong></td>
|
| 524 |
-
<td align="center">87.1</td>
|
| 525 |
-
<td align="center">86.1</td>
|
| 526 |
-
<td align="center">84.9</td>
|
| 527 |
-
</tr>
|
| 528 |
-
<tr>
|
| 529 |
-
<td align="center">MMLU-pro</td>
|
| 530 |
-
<td align="center">EM</td>
|
| 531 |
-
<td align="center">5-shot</td>
|
| 532 |
-
<td align="center"><strong>69.2</strong></td>
|
| 533 |
-
<td align="center">60.6</td>
|
| 534 |
-
<td align="center">62.8</td>
|
| 535 |
-
<td align="center">63.5</td>
|
| 536 |
-
</tr>
|
| 537 |
-
<tr>
|
| 538 |
-
<td align="center">MMLU-redux-2.0</td>
|
| 539 |
-
<td align="center">EM</td>
|
| 540 |
-
<td align="center">5-shot</td>
|
| 541 |
-
<td align="center"><strong>90.2</strong></td>
|
| 542 |
-
<td align="center">89.5</td>
|
| 543 |
-
<td align="center">87.8</td>
|
| 544 |
-
<td align="center">88.2</td>
|
| 545 |
-
</tr>
|
| 546 |
-
<tr>
|
| 547 |
-
<td align="center">SimpleQA</td>
|
| 548 |
-
<td align="center">Correct</td>
|
| 549 |
-
<td align="center">5-shot</td>
|
| 550 |
-
<td align="center"><strong>35.3</strong></td>
|
| 551 |
-
<td align="center">26.5</td>
|
| 552 |
-
<td align="center">10.3</td>
|
| 553 |
-
<td align="center">23.7</td>
|
| 554 |
-
</tr>
|
| 555 |
-
<tr>
|
| 556 |
-
<td align="center">TriviaQA</td>
|
| 557 |
-
<td align="center">EM</td>
|
| 558 |
-
<td align="center">5-shot</td>
|
| 559 |
-
<td align="center"><strong>85.1</strong></td>
|
| 560 |
-
<td align="center">84.1</td>
|
| 561 |
-
<td align="center">76.0</td>
|
| 562 |
-
<td align="center">79.3</td>
|
| 563 |
-
</tr>
|
| 564 |
-
<tr>
|
| 565 |
-
<td align="center">GPQA-Diamond</td>
|
| 566 |
-
<td align="center">Avg@8</td>
|
| 567 |
-
<td align="center">5-shot</td>
|
| 568 |
-
<td align="center">48.1</td>
|
| 569 |
-
<td align="center"><strong>50.5</strong></td>
|
| 570 |
-
<td align="center">40.8</td>
|
| 571 |
-
<td align="center">49.4</td>
|
| 572 |
-
</tr>
|
| 573 |
-
<tr>
|
| 574 |
-
<td align="center">SuperGPQA</td>
|
| 575 |
-
<td align="center">EM</td>
|
| 576 |
-
<td align="center">5-shot</td>
|
| 577 |
-
<td align="center"><strong>44.7</strong></td>
|
| 578 |
-
<td align="center">39.2</td>
|
| 579 |
-
<td align="center">34.2</td>
|
| 580 |
-
<td align="center">38.8</td>
|
| 581 |
-
</tr>
|
| 582 |
-
<tr>
|
| 583 |
-
<td align="center" colspan="7"><strong>Coding Tasks</strong></td>
|
| 584 |
-
</tr>
|
| 585 |
-
<tr>
|
| 586 |
-
<td align="center">LiveCodeBench v6</td>
|
| 587 |
-
<td align="center">Pass@1</td>
|
| 588 |
-
<td align="center">1-shot</td>
|
| 589 |
-
<td align="center"><strong>26.3</strong></td>
|
| 590 |
-
<td align="center">22.9</td>
|
| 591 |
-
<td align="center">21.1</td>
|
| 592 |
-
<td align="center">25.1</td>
|
| 593 |
-
</tr>
|
| 594 |
-
<tr>
|
| 595 |
-
<td align="center">EvalPlus</td>
|
| 596 |
-
<td align="center">Pass@1</td>
|
| 597 |
-
<td align="center">-</td>
|
| 598 |
-
<td align="center"><strong>80.3</strong></td>
|
| 599 |
-
<td align="center">65.6</td>
|
| 600 |
-
<td align="center">66.0</td>
|
| 601 |
-
<td align="center">65.5</td>
|
| 602 |
-
</tr>
|
| 603 |
-
<tr>
|
| 604 |
-
<td align="center" colspan="7"><strong>Mathematics Tasks</strong></td>
|
| 605 |
-
</tr>
|
| 606 |
-
<tr>
|
| 607 |
-
<td align="center">MATH</td>
|
| 608 |
-
<td align="center">EM</td>
|
| 609 |
-
<td align="center">4-shot</td>
|
| 610 |
-
<td align="center"><strong>70.2</strong></td>
|
| 611 |
-
<td align="center">60.1</td>
|
| 612 |
-
<td align="center">61.0</td>
|
| 613 |
-
<td align="center">63.0</td>
|
| 614 |
-
</tr>
|
| 615 |
-
<tr>
|
| 616 |
-
<td align="center">GSM8k</td>
|
| 617 |
-
<td align="center">EM</td>
|
| 618 |
-
<td align="center">8-shot</td>
|
| 619 |
-
<td align="center"><strong>92.1</strong></td>
|
| 620 |
-
<td align="center">91.7</td>
|
| 621 |
-
<td align="center">90.4</td>
|
| 622 |
-
<td align="center">86.3</td>
|
| 623 |
-
</tr>
|
| 624 |
-
<tr>
|
| 625 |
-
<td align="center" colspan="7"><strong>Chinese Tasks</strong></td>
|
| 626 |
-
</tr>
|
| 627 |
-
<tr>
|
| 628 |
-
<td align="center">C-Eval</td>
|
| 629 |
-
<td align="center">EM</td>
|
| 630 |
-
<td align="center">5-shot</td>
|
| 631 |
-
<td align="center"><strong>92.5</strong></td>
|
| 632 |
-
<td align="center">90.0</td>
|
| 633 |
-
<td align="center">90.9</td>
|
| 634 |
-
<td align="center">80.9</td>
|
| 635 |
-
</tr>
|
| 636 |
-
<tr>
|
| 637 |
-
<td align="center">CSimpleQA</td>
|
| 638 |
-
<td align="center">Correct</td>
|
| 639 |
-
<td align="center">5-shot</td>
|
| 640 |
-
<td align="center"><strong>77.6</strong></td>
|
| 641 |
-
<td align="center">72.1</td>
|
| 642 |
-
<td align="center">50.5</td>
|
| 643 |
-
<td align="center">53.5</td>
|
| 644 |
-
</tr>
|
| 645 |
-
</tbody>
|
| 646 |
-
</table>
|
| 647 |
-
</div>
|
| 648 |
-
<sup>
|
| 649 |
-
• We only evaluate open-source pretrained models in this work. We report results for Qwen2.5-72B because the base checkpoint for Qwen3-235B-A22B was not open-sourced at the time of our study.
|
| 650 |
-
</sup><br/><sup>
|
| 651 |
-
• All models are evaluated using the same evaluation protocol.
|
| 652 |
-
|
| 653 |
-
</sup>
|
| 654 |
-
|
| 655 |
-
|
| 656 |
-
## 4. Deployment
|
| 657 |
-
> [!Note]
|
| 658 |
-
> You can access Kimi K2's API on https://platform.moonshot.ai , we provide OpenAI/Anthropic-compatible API for you.
|
| 659 |
-
>
|
| 660 |
-
> The Anthropic-compatible API maps temperature by `real_temperature = request_temperature * 0.6` for better compatible with existing applications.
|
| 661 |
-
|
| 662 |
-
Our model checkpoints are stored in the block-fp8 format, you can find it on [Huggingface](https://huggingface.co/moonshotai/Kimi-K2-Instruct).
|
| 663 |
-
|
| 664 |
-
Currently, Kimi-K2 is recommended to run on the following inference engines:
|
| 665 |
-
|
| 666 |
-
* vLLM
|
| 667 |
-
* SGLang
|
| 668 |
-
* KTransformers
|
| 669 |
-
* TensorRT-LLM
|
| 670 |
-
|
| 671 |
-
Deployment examples for vLLM and SGLang can be found in the [Model Deployment Guide](docs/deploy_guidance.md).
|
| 672 |
-
|
| 673 |
-
---
|
| 674 |
-
|
| 675 |
-
## 5. Model Usage
|
| 676 |
-
|
| 677 |
-
### Chat Completion
|
| 678 |
-
|
| 679 |
-
Once the local inference service is up, you can interact with it through the chat endpoint:
|
| 680 |
-
|
| 681 |
-
```python
|
| 682 |
-
def simple_chat(client: OpenAI, model_name: str):
|
| 683 |
-
messages = [
|
| 684 |
-
{"role": "system", "content": "You are Kimi, an AI assistant created by Moonshot AI."},
|
| 685 |
-
{"role": "user", "content": [{"type": "text", "text": "Please give a brief self-introduction."}]},
|
| 686 |
-
]
|
| 687 |
-
response = client.chat.completions.create(
|
| 688 |
-
model=model_name,
|
| 689 |
-
messages=messages,
|
| 690 |
-
stream=False,
|
| 691 |
-
temperature=0.6,
|
| 692 |
-
max_tokens=256
|
| 693 |
-
)
|
| 694 |
-
print(response.choices[0].message.content)
|
| 695 |
-
```
|
| 696 |
-
|
| 697 |
-
> [!NOTE]
|
| 698 |
-
> The recommended temperature for Kimi-K2-Instruct is `temperature = 0.6`.
|
| 699 |
-
> If no special instructions are required, the system prompt above is a good default.
|
| 700 |
-
|
| 701 |
-
---
|
| 702 |
-
|
| 703 |
-
### Tool Calling
|
| 704 |
-
|
| 705 |
-
Kimi-K2-Instruct has strong tool-calling capabilities.
|
| 706 |
-
To enable them, you need to pass the list of available tools in each request, then the model will autonomously decide when and how to invoke them.
|
| 707 |
-
|
| 708 |
-
The following example demonstrates calling a weather tool end-to-end:
|
| 709 |
-
|
| 710 |
-
```python
|
| 711 |
-
# Your tool implementation
|
| 712 |
-
def get_weather(city: str) -> dict:
|
| 713 |
-
return {"weather": "Sunny"}
|
| 714 |
-
|
| 715 |
-
# Tool schema definition
|
| 716 |
-
tools = [{
|
| 717 |
-
"type": "function",
|
| 718 |
-
"function": {
|
| 719 |
-
"name": "get_weather",
|
| 720 |
-
"description": "Retrieve current weather information. Call this when the user asks about the weather.",
|
| 721 |
-
"parameters": {
|
| 722 |
-
"type": "object",
|
| 723 |
-
"required": ["city"],
|
| 724 |
-
"properties": {
|
| 725 |
-
"city": {
|
| 726 |
-
"type": "string",
|
| 727 |
-
"description": "Name of the city"
|
| 728 |
-
}
|
| 729 |
-
}
|
| 730 |
-
}
|
| 731 |
-
}
|
| 732 |
-
}]
|
| 733 |
-
|
| 734 |
-
# Map tool names to their implementations
|
| 735 |
-
tool_map = {
|
| 736 |
-
"get_weather": get_weather
|
| 737 |
-
}
|
| 738 |
-
|
| 739 |
-
def tool_call_with_client(client: OpenAI, model_name: str):
|
| 740 |
-
messages = [
|
| 741 |
-
{"role": "system", "content": "You are Kimi, an AI assistant created by Moonshot AI."},
|
| 742 |
-
{"role": "user", "content": "What's the weather like in Beijing today? Use the tool to check."}
|
| 743 |
-
]
|
| 744 |
-
finish_reason = None
|
| 745 |
-
while finish_reason is None or finish_reason == "tool_calls":
|
| 746 |
-
completion = client.chat.completions.create(
|
| 747 |
-
model=model_name,
|
| 748 |
-
messages=messages,
|
| 749 |
-
temperature=0.6,
|
| 750 |
-
tools=tools, # tool list defined above
|
| 751 |
-
tool_choice="auto"
|
| 752 |
-
)
|
| 753 |
-
choice = completion.choices[0]
|
| 754 |
-
finish_reason = choice.finish_reason
|
| 755 |
-
if finish_reason == "tool_calls":
|
| 756 |
-
messages.append(choice.message)
|
| 757 |
-
for tool_call in choice.message.tool_calls:
|
| 758 |
-
tool_call_name = tool_call.function.name
|
| 759 |
-
tool_call_arguments = json.loads(tool_call.function.arguments)
|
| 760 |
-
tool_function = tool_map[tool_call_name]
|
| 761 |
-
tool_result = tool_function(**tool_call_arguments)
|
| 762 |
-
print("tool_result:", tool_result)
|
| 763 |
-
|
| 764 |
-
messages.append({
|
| 765 |
-
"role": "tool",
|
| 766 |
-
"tool_call_id": tool_call.id,
|
| 767 |
-
"name": tool_call_name,
|
| 768 |
-
"content": json.dumps(tool_result)
|
| 769 |
-
})
|
| 770 |
-
print("-" * 100)
|
| 771 |
-
print(choice.message.content)
|
| 772 |
-
```
|
| 773 |
-
|
| 774 |
-
The `tool_call_with_client` function implements the pipeline from user query to tool execution.
|
| 775 |
-
This pipeline requires the inference engine to support Kimi-K2’s native tool-parsing logic.
|
| 776 |
-
For streaming output and manual tool-parsing, see the [Tool Calling Guide](docs/tool_call_guidance.md).
|
| 777 |
-
|
| 778 |
-
---
|
| 779 |
-
|
| 780 |
-
## 6. License
|
| 781 |
-
|
| 782 |
-
Both the code repository and the model weights are released under the [Modified MIT License](LICENSE).
|
| 783 |
-
|
| 784 |
-
---
|
| 785 |
-
|
| 786 |
-
## 7. Contact Us
|
| 787 |
-
|
| 788 |
-
If you have any questions, please reach out at [[email protected]](mailto:[email protected]).
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model:
|
| 3 |
+
- moonshotai/Kimi-K2-Instruct
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
+
[<img src="https://raw.githubusercontent.com/csabakecskemeti/devquasar/main/dq_logo_black-transparent.png" width="200"/>](https://devquasar.com)
|
| 8 |
|
| 9 |
+
'Make knowledge free for everyone'
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
+
BF16 version of: [moonshotai/Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)
|
| 12 |
+
<a href='https://ko-fi.com/L4L416YX7C' target='_blank'><img height='36' style='border:0px;height:36px;' src='https://storage.ko-fi.com/cdn/kofi6.png?v=6' border='0' alt='Buy Me a Coffee at ko-fi.com' /></a>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|