Post
248
RuQualBench 🐸
I created a benchmark to evaluate the quality of the Russian language in LLMs. Details:
- A set of 100 (default)/250/500 questions on general chat/creative writing domains.
- LLM-as-a-Judge, but with clear criteria for marking answers.
- Focuses on errors that are typical of LLMs in Russian, such as mixed grammatical genders, characters from other alphabets, and made-up words.
- Everything is under an open license!
Analysis of results:
- The best models are still closed source ones, such as Sonnet 4.5, Gemini, and GPT-4o. However, some open models are very close.
- GPT-5 is terrible. I thought it would be better.
- Of the open models, Gemma-3-27b-it and Vistral-24B are unrivaled.
- Ruadapt significantly reduces errors compared to Qwen.
- Qwen3 and GPT-oss are very bad. They're even worse than I expected.
- Qwen3-Next is better than Qwen3. It seems like they added Russian to train dataset.
- DeepSeek V3 has few errors, but V3.2-Exp is almost twice as bad.
kristaller486/RuQualBench, https://github.com/kristaller486/RuQualBench
I created a benchmark to evaluate the quality of the Russian language in LLMs. Details:
- A set of 100 (default)/250/500 questions on general chat/creative writing domains.
- LLM-as-a-Judge, but with clear criteria for marking answers.
- Focuses on errors that are typical of LLMs in Russian, such as mixed grammatical genders, characters from other alphabets, and made-up words.
- Everything is under an open license!
Analysis of results:
- The best models are still closed source ones, such as Sonnet 4.5, Gemini, and GPT-4o. However, some open models are very close.
- GPT-5 is terrible. I thought it would be better.
- Of the open models, Gemma-3-27b-it and Vistral-24B are unrivaled.
- Ruadapt significantly reduces errors compared to Qwen.
- Qwen3 and GPT-oss are very bad. They're even worse than I expected.
- Qwen3-Next is better than Qwen3. It seems like they added Russian to train dataset.
- DeepSeek V3 has few errors, but V3.2-Exp is almost twice as bad.
kristaller486/RuQualBench, https://github.com/kristaller486/RuQualBench