|
|
--- |
|
|
license: apache-2.0 |
|
|
license_link: https://huggingface.co/skt/A.X-3.1/blob/main/LICENSE |
|
|
language: |
|
|
- en |
|
|
- ko |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
model_id: skt/A.X-3.1 |
|
|
developers: SKT AI Model Lab |
|
|
model-index: |
|
|
- name: A.X-3.1 |
|
|
results: |
|
|
- task: |
|
|
type: generate_until |
|
|
name: mmlu |
|
|
dataset: |
|
|
name: mmlu (chat CoT) |
|
|
type: hails/mmlu_no_train |
|
|
metrics: |
|
|
- type: exact_match |
|
|
value: 75.1 |
|
|
name: exact_match |
|
|
- task: |
|
|
type: generate_until |
|
|
name: kmmlu |
|
|
dataset: |
|
|
name: kmmlu (chat CoT) |
|
|
type: HAERAE-HUB/KMMLU |
|
|
metrics: |
|
|
- type: exact_match |
|
|
value: 69.2 |
|
|
name: exact_match |
|
|
--- |
|
|
|
|
|
# A.X 3.1 |
|
|
|
|
|
<div align="center"> |
|
|
<img src="./assets/A.X_from_scratch_logo_ko_4x3.png" alt="A.X Logo" width="300"/> |
|
|
</div> |
|
|
<p align="center"> <a href="https://huggingface.co/collections/skt/ax-3-686b288b3b05e1234f3f4c73">🤗 Models</a> | <a href="https://github.com/SKT-AI/A.X-3">🖥️ Github</a> </p> |
|
|
|
|
|
## A.X 3.1 Highlights |
|
|
|
|
|
SK Telecom released **A.X 3.1** (pronounced "A dot X"), a large language model (LLM) optimized for Korean-language understanding and enterprise deployment, on July 24, 2025. |
|
|
This sovereign AI model was developed entirely in-house by SKT, encompassing model architecture, data curation, and training, all carried out on SKT’s proprietary supercomputing infrastructure, TITAN. |
|
|
The model was trained from scratch on a high-quality multilingual corpus comprising **2.1 trillion tokens**, with a primary focus on the Korean language. |
|
|
|
|
|
- **Authentic Korean Sovereign AI**: A.X 3.1 was trained on a high-quality multilingual dataset—fully curated in-house—using SKT’s proprietary GPU infrastructure. |
|
|
- **Highly Efficient Multilingual LLM**: A.X 3.1 demonstrates superior performance among Korean LLMs, despite its relatively compact training size of 2.1 trillion tokens. |
|
|
- **Superior Korean Proficiency**: A.X 3.1 achieved a score of **69.2** on the [KMMLU](https://huggingface.co/datasets/HAERAE-HUB/KMMLU): the leading benchmark for Korean-language evaluation and a Korean-specific adaptation of MMLU, outperforming other Korean-specified models. |
|
|
- **Deep Korean Understanding**: A.X 3.1 obtained **77.4** on the [CLIcK](https://huggingface.co/datasets/EunsuKim/CLIcK): a benchmark for Korean cultural and contextual comprehension, outperforming other open-source models. |
|
|
- **Efficient Token Usage**: A.X 3.1 requires approximately 33% fewer tokens than GPT-4o to process equivalent Korean inputs, facilitating more cost-effective and computationally efficient inference. |
|
|
- **Long-Context Handling**: A.X 3.1 supports up to **32,768 tokens** natively, and up to **131,072 tokens** by applying YaRN. |
|
|
|
|
|
|
|
|
## Core Technologies |
|
|
|
|
|
A.X 3.1 represents **an efficient sovereign AI model**, developed end-to-end by SKT, encompassing model architecture, data curation, infrastructure deployment, and optimization. |
|
|
|
|
|
### Model Architecture Specs |
|
|
|
|
|
<table><thead> |
|
|
<tr> |
|
|
<th>Model</th> |
|
|
<th># Params</th> |
|
|
<th># Layers</th> |
|
|
<th># KV-Heads</th> |
|
|
<th>Hidden Dim</th> |
|
|
<th>FFN Dim</th> |
|
|
</tr> |
|
|
<tr> |
|
|
<th>A.X 3.1</th> |
|
|
<th>34B</th> |
|
|
<th>48</th> |
|
|
<th>8</th> |
|
|
<th>8192</th> |
|
|
<th>21824</th> |
|
|
</tr> |
|
|
</thead> |
|
|
</table> |
|
|
|
|
|
### High-Quality Data Pipeline & Strategic Mixture |
|
|
|
|
|
- We collected and curated a training dataset comprising 20 trillion tokens sourced from diverse domains. |
|
|
- The entire dataset was processed through SKT’s proprietary data pipeline, incorporating synthetic data generation and comprehensive quality filtering. |
|
|
- For training A.X 3.1, a total of **2.1 trillion tokens** were utilized, comprising a Korean-focused multilingual corpus. |
|
|
|
|
|
|
|
|
## Benchmark Results |
|
|
|
|
|
### Model Performance |
|
|
|
|
|
<table> |
|
|
<caption style="text-align:left; caption-side:bottom">* self-reported score</caption> |
|
|
<thead> |
|
|
<tr> |
|
|
<th></th> |
|
|
<th></th> |
|
|
<th>A.X 3.1</th> |
|
|
<th>EXAONE-3.5-32B</th> |
|
|
<th>Kanana-flag-32.5B</th> |
|
|
<th>Gemma-3-27B</th> |
|
|
<th>Qwen2.5-32B</th> |
|
|
</tr></thead> |
|
|
<tbody> |
|
|
<tr> |
|
|
<td rowspan="5">Knowledge</td> |
|
|
<td>KMMLU</td> |
|
|
<td>69.73</td> |
|
|
<td>57.17</td> |
|
|
<td>64.19*</td> |
|
|
<td>59.45</td> |
|
|
<td>61.93</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>KMMLU-pro</td> |
|
|
<td>54.89</td> |
|
|
<td>45.39</td> |
|
|
<td>-</td> |
|
|
<td>50.43</td> |
|
|
<td>52.34</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>KMMLU-redux</td> |
|
|
<td>62.66</td> |
|
|
<td>48.32</td> |
|
|
<td>-</td> |
|
|
<td>54.85</td> |
|
|
<td>52.15</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>Click (chat CoT)</td> |
|
|
<td>77.09</td> |
|
|
<td>69.42</td> |
|
|
<td>-</td> |
|
|
<td>71.03</td> |
|
|
<td>68.17</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MMLU</td> |
|
|
<td>75.20</td> |
|
|
<td>77.1</td> |
|
|
<td>81.08*</td> |
|
|
<td>82.35</td> |
|
|
<td>83.4</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">General</td> |
|
|
<td>Ko-MT-bench</td> |
|
|
<td>83.06</td> |
|
|
<td>80.19</td> |
|
|
<td>80.58*</td> |
|
|
<td>85.5</td> |
|
|
<td>72.88</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MT-bench</td> |
|
|
<td>84.19</td> |
|
|
<td>85.09</td> |
|
|
<td>83.56*</td> |
|
|
<td>84.38</td> |
|
|
<td>87.31</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">IF</td> |
|
|
<td>Ko-IFEval</td> |
|
|
<td>75.29</td> |
|
|
<td>68.67</td> |
|
|
<td>-</td> |
|
|
<td>74.4</td> |
|
|
<td>73.24</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>IFEval</td> |
|
|
<td>87.11</td> |
|
|
<td>82.67</td> |
|
|
<td>85.6*</td> |
|
|
<td>82.45</td> |
|
|
<td>82.27</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">Math<br> </td> |
|
|
<td>HRM8K</td> |
|
|
<td>45.53</td> |
|
|
<td>36.3</td> |
|
|
<td>-</td> |
|
|
<td>48</td> |
|
|
<td>41.29</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MATH</td> |
|
|
<td>75.40</td> |
|
|
<td>61.64</td> |
|
|
<td>57.82*</td> |
|
|
<td>80.72</td> |
|
|
<td>73.26</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="3">Code<br> <br> </td> |
|
|
<td>HumanEval+</td> |
|
|
<td>75.00</td> |
|
|
<td>77.44</td> |
|
|
<td>77.44*</td> |
|
|
<td>78.66</td> |
|
|
<td>82.32</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MBPP+</td> |
|
|
<td>70.90</td> |
|
|
<td>65.87</td> |
|
|
<td>69.84*</td> |
|
|
<td>74.07</td> |
|
|
<td>73.81</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>LiveCodeBench</td> |
|
|
<td>23.34</td> |
|
|
<td>17.2</td> |
|
|
<td>-</td> |
|
|
<td>30.55</td> |
|
|
<td>26.9</td> |
|
|
</tr> |
|
|
</tbody></table> |
|
|
|
|
|
|
|
|
### Lightweight Model Performance |
|
|
|
|
|
<table><thead> |
|
|
<tr> |
|
|
<th colspan="2">Benchmarks</th> |
|
|
<th>A.X 3.1 Light</th> |
|
|
<th>Kanana-1.5-8B</th> |
|
|
<th>EXAONE-3.5-7.8B</th> |
|
|
<th>Qwen2.5-7B</th> |
|
|
<th>Qwen3-8B<br>(w/o reasoning)</th> |
|
|
</tr></thead> |
|
|
<tbody> |
|
|
<tr> |
|
|
<td rowspan="6">Knowledge</td> |
|
|
<td>KMMLU</td> |
|
|
<td>61.70</td> |
|
|
<td>48.28</td> |
|
|
<td>53.76</td> |
|
|
<td>49.56</td> |
|
|
<td>63.53</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>KMMLU-pro</td> |
|
|
<td>45.54</td> |
|
|
<td>37.63</td> |
|
|
<td>40.11</td> |
|
|
<td>38.87</td> |
|
|
<td>50.71</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>KMMLU-redux</td> |
|
|
<td>52.34</td> |
|
|
<td>35.33</td> |
|
|
<td>42.21</td> |
|
|
<td>38.58</td> |
|
|
<td>55.74</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>CLIcK</td> |
|
|
<td>71.22</td> |
|
|
<td>61.30</td> |
|
|
<td>64.11</td> |
|
|
<td>58.30</td> |
|
|
<td>63.31</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>KoBALT</td> |
|
|
<td>27.43</td> |
|
|
<td>23.14</td> |
|
|
<td>21.71</td> |
|
|
<td>21.57</td> |
|
|
<td>26.57</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MMLU</td> |
|
|
<td>66.95</td> |
|
|
<td>68.82</td> |
|
|
<td>72.20</td> |
|
|
<td>75.40</td> |
|
|
<td>82.89</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">General</td> |
|
|
<td>Ko-MT-Bench</td> |
|
|
<td>78.56</td> |
|
|
<td>76.30</td> |
|
|
<td>81.06</td> |
|
|
<td>61.31</td> |
|
|
<td>64.06</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MT-Bench</td> |
|
|
<td>74.38</td> |
|
|
<td>77.60</td> |
|
|
<td>83.50</td> |
|
|
<td>79.37</td> |
|
|
<td>65.69</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">Instruction<br>Following</td> |
|
|
<td>Ko-IFEval</td> |
|
|
<td>70.04</td> |
|
|
<td>69.96</td> |
|
|
<td>65.01</td> |
|
|
<td>60.73</td> |
|
|
<td>73.39</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>IFEval</td> |
|
|
<td>79.86</td> |
|
|
<td>80.11</td> |
|
|
<td>82.61</td> |
|
|
<td>76.73</td> |
|
|
<td>85.38</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">Math</td> |
|
|
<td>HRM8K</td> |
|
|
<td>41.70</td> |
|
|
<td>30.87</td> |
|
|
<td>31.88</td> |
|
|
<td>35.13</td> |
|
|
<td>52.50</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MATH</td> |
|
|
<td>70.14</td> |
|
|
<td>59.28</td> |
|
|
<td>63.20</td> |
|
|
<td>65.58</td> |
|
|
<td>71.48</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2">Code<br></td> |
|
|
<td>HumanEval+</td> |
|
|
<td>73.78</td> |
|
|
<td>76.83</td> |
|
|
<td>76.83</td> |
|
|
<td>74.39</td> |
|
|
<td>77.44</td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td>MBPP+</td> |
|
|
<td>61.64</td> |
|
|
<td>67.99</td> |
|
|
<td>64.29</td> |
|
|
<td>68.50</td> |
|
|
<td>62.17</td> |
|
|
</tr> |
|
|
</tbody></table> |
|
|
|
|
|
## 🚀 Quickstart |
|
|
|
|
|
### with HuggingFace Transformers |
|
|
|
|
|
- `transformers>=4.46.0` or the latest version is required to use `skt/A.X-3.1` |
|
|
```bash |
|
|
pip install transformers>=4.46.0 |
|
|
``` |
|
|
|
|
|
#### Example Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
model_name = "skt/A.X-3.1" |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_name, |
|
|
torch_dtype=torch.bfloat16, |
|
|
device_map="auto", |
|
|
) |
|
|
model.eval() |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
|
|
messages = [ |
|
|
{"role": "system", "content": "당신은 사용자가 제공하는 영어 문장들을 한국어로 번역하는 AI 전문가입니다."}, |
|
|
{"role": "user", "content": "The first human went into space and orbited the Earth on April 12, 1961."}, |
|
|
] |
|
|
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device) |
|
|
|
|
|
with torch.no_grad(): |
|
|
output = model.generate( |
|
|
input_ids, |
|
|
max_new_tokens=128, |
|
|
do_sample=False, |
|
|
) |
|
|
|
|
|
len_input_prompt = len(input_ids[0]) |
|
|
response = tokenizer.decode(output[0][len_input_prompt:], skip_special_tokens=True) |
|
|
print(response) |
|
|
# Output: |
|
|
# 우주에서 인간이 처음으로 지구 궤도를 돈 날은 1961년 4월 12일입니다. |
|
|
``` |
|
|
|
|
|
### with vLLM |
|
|
|
|
|
- `vllm>=v0.6.4.post1` or the latest version is required to use tool-use feature |
|
|
```bash |
|
|
pip install vllm>=v0.6.4.post1 |
|
|
# if you don't want to activate tool-use feature, just commenting out below vLLM option |
|
|
VLLM_OPTION="--enable-auto-tool-choice --tool-call-parser hermes" |
|
|
vllm serve skt/A.X-3.1 $VLLM_OPTION |
|
|
``` |
|
|
|
|
|
#### Example Usage |
|
|
|
|
|
```python |
|
|
from openai import OpenAI |
|
|
|
|
|
def call(messages, model): |
|
|
completion = client.chat.completions.create( |
|
|
model=model, |
|
|
messages=messages, |
|
|
) |
|
|
print(completion.choices[0].message) |
|
|
|
|
|
client = OpenAI( |
|
|
base_url="http://localhost:8000/v1", |
|
|
api_key="api_key" |
|
|
) |
|
|
model = "skt/A.X-3.1" |
|
|
messages = [{"role": "user", "content": "에어컨 여름철 적정 온도는? 한줄로 답변해줘"}] |
|
|
call(messages, model) |
|
|
# Output: |
|
|
# 여름철 에어컨 적정 온도는 24~26도입니다. |
|
|
|
|
|
messages = [{"role": "user", "content": "What is the appropriate temperature for air conditioning in summer? Respond in a single sentence."}] |
|
|
call(messages, model) |
|
|
# Output: |
|
|
# The appropriate temperature for air conditioning in summer is around 78°F (26°C). |
|
|
``` |
|
|
|
|
|
#### Examples for tool-use |
|
|
```python |
|
|
from openai import OpenAI |
|
|
|
|
|
|
|
|
def call(messages, model): |
|
|
completion = client.chat.completions.create( |
|
|
model=model, |
|
|
messages=messages, |
|
|
tools=tools |
|
|
) |
|
|
print(completion.choices[0].message) |
|
|
|
|
|
|
|
|
client = OpenAI( |
|
|
base_url="http://localhost:8000/v1", |
|
|
api_key="api_key" |
|
|
) |
|
|
model = "skt/A.X-3.1" |
|
|
|
|
|
calculate_discount = { |
|
|
"type": "function", |
|
|
"function": { |
|
|
"name": "calculate_discount", |
|
|
"description": "원가격과 할인율(퍼센트 단위)을 입력받아 할인된 가격을계산한다.", |
|
|
"parameters": { |
|
|
"type": "object", |
|
|
"properties": { |
|
|
"original_price": { |
|
|
"type": "number", |
|
|
"description": "상품의 원래 가격" |
|
|
}, |
|
|
"discount_percentage": { |
|
|
"type": "number", |
|
|
"description": "적용할 할인율" |
|
|
} |
|
|
}, |
|
|
"required": ["original_price", "discount_percentage"] |
|
|
} |
|
|
} |
|
|
} |
|
|
get_exchange_rate = { |
|
|
"type": "function", |
|
|
"function": { |
|
|
"name": "get_exchange_rate", |
|
|
"description": "두 통화 간의 환율을 가져온다.", |
|
|
"parameters": { |
|
|
"type": "object", |
|
|
"properties": { |
|
|
"base_currency": { |
|
|
"type": "string", |
|
|
"description": "The currency to convert from." |
|
|
}, |
|
|
"target_currency": { |
|
|
"type": "string", |
|
|
"description": "The currency to convert to." |
|
|
} |
|
|
}, |
|
|
"required": ["base_currency", "target_currency"] |
|
|
} |
|
|
} |
|
|
} |
|
|
tools = [calculate_discount, get_exchange_rate] |
|
|
|
|
|
### Slot filling ### |
|
|
messages = [{"role": "user", "content": "우리가 뭘 사야되는데 원가가 57600원인데 직원할인 받으면 얼마야?"}] |
|
|
call(messages, model) |
|
|
# Output: |
|
|
# ChatCompletionMessage(content='직원 할인율이 몇 퍼센트인지 알려주신다면 할인된 가격을 계산할 수 있습니다. 할인율이 몇 퍼센트인지 알려주실 수 있나요?', role='assistant', tool_calls=[]) |
|
|
|
|
|
|
|
|
### Function calling ### |
|
|
messages = [ |
|
|
{"role": "user", "content": "우리가 뭘 사야되는데 원가가 57600원인데 직원할인 받으면 얼마야?"}, |
|
|
{"role": "assistant", "content": "직원 할인율이 몇 퍼센트인지 알려주신다면 할인된 가격을 계산할 수 있습니다. 할인율이 몇 퍼센트인지 알려주실 수 있나요?"}, |
|
|
{"role": "user", "content": "15% 할인 받을 수 있어."}, |
|
|
] |
|
|
call(messages, model) |
|
|
# Output: |
|
|
# ChatCompletionMessage(content=None, role='assistant', tool_calls=[ChatCompletionMessageToolCall(id='chatcmpl-tool-cb9e827f752d4725abc94377223b2b0f', function=Function(arguments='{"original_price": 57600, "discount_percentage": 15}', name='calculate_discount'), type='function')]) |
|
|
|
|
|
|
|
|
### Completion ### |
|
|
messages = [ |
|
|
{"role": "user", "content": "우리가 뭘 사야되는데 원가가 57600원인데 직원할인 받으면 얼마야?"}, |
|
|
{"role": "assistant", "content": "직원 할인율이 몇 퍼센트인지 알려주신다면 할인된 가격을 계산할 수 있습니다. 할인율이 몇 퍼센트인지 알려주실 수 있나요?"}, |
|
|
{"role": "user", "content": "15% 할인 받을 수 있어."}, |
|
|
{"role": "tool", "tool_call_id": "random_id", "name": "calculate_discount", "content": "{\"original_price\": 57600, \"discount_percentage\": 15, \"discounted_price\": 48960.0}"} |
|
|
] |
|
|
call(messages, model) |
|
|
# Output: |
|
|
# ChatCompletionMessage(content='직원 할인을 받으면 57600원의 상품은 15% 할인을 받아 48960원이 됩니다.', role='assistant', tool_calls=[]) |
|
|
``` |
|
|
|
|
|
### Extend supported token length |
|
|
|
|
|
The `config.json` file of A.X 3.1 uploaded to HuggingFace is configured for maximum token lengths of 32,768. You can simply handle up to 131,072 tokens by modifying `rope_scaling` field in `config.json` file into the following parameters: |
|
|
|
|
|
``` |
|
|
"rope_scaling": { |
|
|
"type": "yarn", |
|
|
"factor": 4.0, |
|
|
"original_max_position_embeddings": 32768, |
|
|
}, |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
The `A.X 3.1` model is licensed under `Apache License 2.0`. |
|
|
|
|
|
## Citation |
|
|
``` |
|
|
@article{SKTAdotX3.1, |
|
|
title={A.X 3.1}, |
|
|
author={SKT AI Model Lab}, |
|
|
year={2025}, |
|
|
url={https://huggingface.co/skt/A.X-3.1} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Contact |
|
|
|
|
|
- Business & Partnership Contact: [[email protected]]([email protected]) |
