DeBERTa Text Quality Model

This model rates the quality of English text for AI learning. Input a text string, and it outputs a numeric quality score reflecting overall informativeness and usefulness.

Performance

On the evaluation set, it achieved:

  • Loss: 0.1408
  • MSE: 0.1408
  • Combined Score: 0.1408
  • Tokens processed during training: 102,398,720

Usage Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "agentlans/deberta-v3-base-quality-v3"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name).to("cuda" if torch.cuda.is_available() else "cpu")

# Higher scores indicate higher text quality.
# The sign of the score has no particular meaning.
# For example, a negative score doesn't necessarily mean that the text is low quality.
def quality(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True).to(model.device)
    with torch.no_grad():
        score = model(**inputs).logits.squeeze().cpu().item()
    return score

print(quality("Your text here."))

Limitations

  • Works best on non-fiction and general-purpose texts.
  • Scores give an overall quality estimate but don’t explain why.
  • The model is large and slow; for faster results with similar accuracy, try agentlans/GIST-all-MiniLM-L6-v2-quality-v3.
  • Check for biases and suitability before use.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • num_epochs: 10.0

Training results

Training Loss Epoch Step Validation Loss Mse Combined Score Input Tokens Seen
0.1635 1.0 10000 0.1854 0.1854 0.1854 10239872
0.1241 2.0 20000 0.1408 0.1408 0.1408 20479744
0.0882 3.0 30000 0.1747 0.1747 0.1747 30719616
0.054 4.0 40000 0.1528 0.1528 0.1528 40959488
0.0372 5.0 50000 0.1480 0.1480 0.1480 51199360
0.0263 6.0 60000 0.1524 0.1524 0.1524 61439232
0.0203 7.0 70000 0.1495 0.1495 0.1495 71679104
0.0135 8.0 80000 0.1482 0.1482 0.1482 81918976
0.0098 9.0 90000 0.1450 0.1450 0.1450 92158848
0.0073 10.0 100000 0.1453 0.1453 0.1453 102398720

Framework versions

  • Transformers 4.51.3
  • Pytorch 2.6.0+cu124
  • Datasets 3.2.0
  • Tokenizers 0.21.0
Downloads last month
11
Safetensors
Model size
184M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for agentlans/deberta-v3-base-quality-v3

Dataset used to train agentlans/deberta-v3-base-quality-v3