mdeberta-v3-base-lite

This model was created through vocabulary pruning of the original microsoft/mdeberta-v3-base model while maintaining full quality for Latin and Cyrillic-based languages.

Supported Languages

• Bulgarian
• English
• French
• German
• Russian
• Spanish
• Swahili
• Turkish
• Vietnamese

Usage

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("rustemgareev/mdeberta-v3-base-lite")
model = AutoModel.from_pretrained("rustemgareev/mdeberta-v3-base-lite")

# Example usage
text = "This is an example text in English."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

Performance Evaluation

Size Comparison

Metric	Original Model	Lite Model	Reduction
Vocabulary Size	250,102 tokens	163,211 tokens	34.74%
Disk Size	1.06 GB	817 MB	23.23%

VRAM Usage Comparison

Estimated using Hugging Face Accelerate Model Estimator.

Metric	Original Model	Lite Model	Reduction
Largest Layer (float32)	735.35 MB	478.16 MB	34.99%
Total Size (float32)	1.04 GB	804.13 MB	22.68%
Training using Adam (Peak vRAM)	4.15 GB	3.14 GB	24.34%

Semantic Similarity Comparison

Evaluation Method: Cosine similarity between embeddings of parallel sentences in different languages, using English as reference.

Test Phrases Used:

• English: "Artificial intelligence learns to understand human languages and helps people communicate."
• Bulgarian: "Изкуственият интелект се учи да разбира човешките езици и помага на хората да общуват."
• French: "L'intelligence artificielle apprend à comprendre les langages humains et aide les gens à communiquer."
• German: "Künstliche Intelligenz lernt, menschliche Sprachen zu verstehen und hilft Menschen bei der Kommunikation."
• Russian: "Искусственный интеллект учится понимать человеческие языки и помогает людям общаться."
• Spanish: "La inteligencia artificial aprende a entender los idiomas humanos y ayuda a las personas a comunicarse."
• Swahili: "Akili ya kisasa inajifunza kuelewa lugha za wanadamu na kusaidia watu kuwasiliana."
• Turkish: "Yapay zeka, insan dillerini anlamayı öğrenir ve insanların iletişim kurmasına yardımcı olur."
• Vietnamese: "Trí tuệ nhân tạo học cách hiểu ngôn ngữ con người và giúp mọi người giao tiếp."

Similarity Results:

Language Pair	Original Similarity	Lite Similarity	Difference
English-Bulgarian	0.9276	0.9276	0.0000
English-French	0.9322	0.9322	0.0000
English-German	0.9178	0.9178	0.0000
English-Russian	0.9335	0.9335	0.0000
English-Spanish	0.9228	0.9228	0.0000
English-Swahili	0.9591	0.9591	0.0000
English-Turkish	0.9450	0.9450	0.0000
English-Vietnamese	0.7955	0.7955	0.0000

License

This model is distributed under the MIT License.