DistilBERT Commission Detection Model
Model Description
This is a fine-tuned DistilBERT model for detecting commission-related content in social media profiles and posts. The model classifies text to identify whether an artist's profile/bio/post content shows they are open or closed for commissions, or if the text is unclear.
Model Details
Model Architecture
- Base Model: distilbert/distilbert-base-uncased
- Model Type: Text Classification
- Language: English
- License: MIT
Training Data
- Sources: Manually scraped profile names, bios, and posts from Bluesky and Twitter by a crowd of furries uploading classifications via a custom extension built specifically to make this dataset
- Dataset: Custom dataset of ~1000 rows and user classifications with an equal amount of artificial data to boost pattern recognition
Performance
Metric | Value |
---|---|
Accuracy | 95.06% |
Precision | 95.13% |
Recall | 95.06% |
F1 Score | 95.08% |
Note: These metrics are not independently verified.
Usage
I recommend a high temperature when inferencing to lower the model's confidence. I use between 1.5 - 3.0.
# Example inference #
from transformers import DistilBertForSequenceClassification, DistilBertTokenizer
import torch
# Load model and tokenizer #
model_name = 'zohfur/distilbert-commissions'
tokenizer = DistilBertTokenizer.from_pretrained(model_name)
model = DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=3)
# Example usage #
example_sentences = [
"Commissions are currently closed.",
"Check my bio for commission status.",
"C*mms 0pen on p-site",
"DM for comms",
"Taking art requests, dm me",
"comm completed for personmcperson, thank you <3",
"open for trades",
"Comms are not open",
"Comms form will be open soon, please check back later",
"~ Furry artist - 25 y.o - he/him - c*mms 0pen: 2/5 - bots dni ~"
]
# Map label integers back to strings #
label_map = {0: 'open', 1: 'closed', 2: 'unclear'}
def predict_with_temperature(model, tokenizer, sentences, temperature=1.5):
# Prepare input #
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
encoded_input = {key: value.to(device) for key, value in encoded_input.items()}
model.to(device)
model.eval()
# Make predictions with temperature scaling #
with torch.no_grad():
outputs = model(**encoded_input)
logits = outputs['logits'] / temperature # Apply temperature scaling #
probabilities = torch.softmax(logits, dim=1)
# Extract predictions and confidence scores #
predicted_class_indices = torch.argmax(probabilities, dim=1)
confidences = torch.max(probabilities, dim=1).values
# Convert to CPU and prepare results #
predictions = {
'sentences': sentences,
'labels': [label_map[idx.item()] for idx in predicted_class_indices],
'confidences': [score.item() for score in confidences]
}
return predictions
def print_predictions(predictions):
"""Print formatted predictions with confidence scores."""
print("\nClassification Results:")
print("=" * 50)
for i, (sentence, label, confidence) in enumerate(zip(
predictions['sentences'],
predictions['labels'],
predictions['confidences']
), 1):
print(f"\n{i}. Sentence: '{sentence}'")
print(f" Predicted Label: {label}")
print(f" Confidence Score: {confidence:.4f}")
# Make predictions with temperature scaling #
predictions = predict_with_temperature(model, tokenizer, example_sentences, temperature=1.5)
# Print results #
print_predictions(predictions)
Limitations and Biases
Limitations
- Language: Only trained on English text
- False Positives: Requires a high temperature to avoid false positives (particularly with the words "open" and "closed")
- Platform Bias: Trained on Bsky and Twitter/X data, might not perform as well on other platforms like FurAffinity or Instagra
Training Details
Training Procedure
- Base Model: DistilBERT base uncased
- Fine-tuning: Finetuned using Huggingface's Trainer, evaluated using Trainer and sklearn.metrics
- Optimization: Wandb hyperparameter sweep using bayers algorithm to reach highest f1 score
Data Preprocessing
- Classifications uploaded voluntarily by crowdsourcing extension users
- Problematic unicode characters cleaned from dataset
- Label encoding for classification
- Class weights computed to adjust weights inversely proportional to class frequencies
Model Card Authors
All credit to original author Zohfur. Base model attributed to distilbert.
Model Card Contact
For questions or concerns about this model, please contact: [[email protected]]
- Downloads last month
- 16
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
๐
Ask for provider support
Model tree for zohfur/distilbert-commissions
Evaluation results
- Accuracy on Scraped Social Media Profiles (Bluesky & Twitter)self-reported0.951
- Precision on Scraped Social Media Profiles (Bluesky & Twitter)self-reported0.951
- Recall on Scraped Social Media Profiles (Bluesky & Twitter)self-reported0.951
- F1 Score on Scraped Social Media Profiles (Bluesky & Twitter)self-reported0.951