Introducing RTEB: A New Standard for Retrieval Evaluation
β’
118
AI & ML interests
Massive Text Embeddings Benchmark
Recent Activity
Papers
HUME: Measuring the Human-Model Performance Gap in Text Embedding Task
Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks
Articles
Organization Card
MTEB is a Python framework for evaluating embeddings and retrieval systems for both text and image. MTEB covers more than 1000 languages and diverse tasks, from classics like classification and clustering to use-case specialized tasks such as legal, code, or healthcare retrieval.
You can get started using mteb, check out our documentation.
| Overview | |
|---|---|
| π Leaderboard | The interactive leaderboard of the benchmark |
| Get Started. | |
| π Get Started | Overview of how to use mteb |
| π€ Defining Models | How to use existing model and define custom ones |
| π Selecting tasks | How to select tasks, benchmarks, splits etc. |
| π Running Evaluation | How to run the evaluations, including cache management, speeding up evaluations etc. |
| π Loading Results | How to load and work with existing model results |
| Overview. | |
| π Tasks | Overview of available tasks |
| π Benchmarks | Overview of available benchmarks |
| π€ Models | Overview of available Models |
| Contributing | |
| π€ Adding a model | How to submit a model to MTEB and to the leaderboard |
| π©βπ» Adding a dataset | How to add a new task/dataset to MTEB |
| π©βπ» Adding a benchmark | How to add a new benchmark to MTEB and to the leaderboard |
| π€ Contributing | How to contribute to MTEB and set it up for development |
This is a collection of MTEB papers (not exhaustive).
-
MMTEB: Massive Multilingual Text Embedding Benchmark
Paper β’ 2502.13595 β’ Published β’ 41 -
MTEB: Massive Text Embedding Benchmark
Paper β’ 2210.07316 β’ Published β’ 6 -
The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
Paper β’ 2406.02396 β’ Published -
Extending the Massive Text Embedding Benchmark to French
Paper β’ 2405.20468 β’ Published β’ 2
This is a collection of MTEB papers (not exhaustive).
-
MMTEB: Massive Multilingual Text Embedding Benchmark
Paper β’ 2502.13595 β’ Published β’ 41 -
MTEB: Massive Text Embedding Benchmark
Paper β’ 2210.07316 β’ Published β’ 6 -
The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
Paper β’ 2406.02396 β’ Published -
Extending the Massive Text Embedding Benchmark to French
Paper β’ 2405.20468 β’ Published β’ 2
A collection of items telated the the MMTEB release
datasets
1,259
mteb/TempReasonL1
Viewer
β’
Updated
β’
20.5k
β’
49
mteb/WinoGrande
Viewer
β’
Updated
β’
7.63k
β’
75
mteb/results
Updated
β’
5.67k
β’
1
mteb/miracl_full
Viewer
β’
Updated
β’
40.6k
β’
38
mteb/CodeRAGStackoverflowPosts
Viewer
β’
Updated
β’
70.6M
β’
15
mteb/XM3600T2IRetrieval
Viewer
β’
Updated
β’
645k
β’
133
mteb/OVENIT2TRetrieval
Viewer
β’
Updated
β’
1.22M
β’
27
mteb/XFlickr30kCoT2IRetrieval
Viewer
β’
Updated
β’
48k
β’
60
mteb/Vidore2ESGReportsRetrieval
Viewer
β’
Updated
β’
9.93k
β’
32
mteb/BUCC
Viewer
β’
Updated
β’
4
β’
40