---
pipeline_tag: translation
language: multilingual
library_name: transformers
base_model:
- FacebookAI/xlm-roberta-large
license: apache-2.0
---
đź“Š Estimating Machine Translation Difficulty
This repository contains the **SENTINELSRC** metric model used for Difficulty Sampling at the [WMT25 General Machine Translation Shared Task](https://www2.statmt.org/wmt25/translation-task.html), and analyzed in our paper **Estimating Machine Translation Difficulty**.
## Usage
To run this model, install the following git repository:
```bash
pip install git+https://github.com/prosho-97/guardians-mt-eval
```
After that, you can use this model within Python in the following way:
```python
from sentinel_metric import download_model, load_from_checkpoint
model_path = download_model("Prosho/sentinel-src-25")
model = load_from_checkpoint(model_path)
data = [
{"src": "Please sign the form."},
{"src": "He spilled the beans, then backpedaled—talk about mixed signals!"}
]
output = model.predict(data, batch_size=8, gpus=1)
```
Output:
```python
# Segment scores
>>> output.scores
[0.5604351758956909, -0.08413456380367279]
# System score
>>> output.system_score
0.23815030604600906
```
Where the higher the output score, the easier it is to translate the input source text.
## Cite this work
This work has been accepted at [EMNLP 2025](https://2025.emnlp.org/). If you use any part, please consider citing our paper as follows:
```bibtex
@misc{proietti2025estimatingmachinetranslationdifficulty,
title={Estimating Machine Translation Difficulty},
author={Lorenzo Proietti and Stefano Perrella and Vilém Zouhar and Roberto Navigli and Tom Kocmi},
year={2025},
eprint={2508.10175},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2508.10175},
}
``` 
