metadata
base_model: cmarkea/distilcamembert-base
datasets:
- CATIE-AQ/frenchSTS
- CATIE-AQ/frenchNLI
language: fr
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
model-index:
- name: CATIE-AQ/distilcamembert-base-embedding
results:
- dataset:
config: default
name: MTEB AlloProfClusteringP2P (default)
revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b
split: test
type: lyon-nlp/alloprof
metrics:
- type: main_score
value: 59.596627257424096
- type: v_measure
value: 59.596627257424096
- type: v_measure_std
value: 4.010558670407883
task:
type: Clustering
- dataset:
config: default
name: MTEB AlloProfClusteringS2S (default)
revision: 392ba3f5bcc8c51f578786c1fc3dae648662cb9b
split: test
type: lyon-nlp/alloprof
metrics:
- type: main_score
value: 47.06111698347941
- type: v_measure
value: 47.06111698347941
- type: v_measure_std
value: 1.5225244073165662
task:
type: Clustering
- dataset:
config: default
name: MTEB AlloprofReranking (default)
revision: 65393d0d7a08a10b4e348135e824f385d420b0fd
split: test
type: lyon-nlp/mteb-fr-reranking-alloprof-s2p
metrics:
- type: main_score
value: 60.04300511045722
- type: map
value: 60.04300511045722
- type: mrr
value: 61.13590413137045
- type: nAUC_map_diff1
value: 39.7665066023792
- type: nAUC_map_max
value: 16.057968603659027
- type: nAUC_map_std
value: 27.01276121764581
- type: nAUC_mrr_diff1
value: 39.35109166795732
- type: nAUC_mrr_max
value: 16.64363630250138
- type: nAUC_mrr_std
value: 26.82808183767377
task:
type: Reranking
- dataset:
config: default
name: MTEB AlloprofRetrieval (default)
revision: fcf295ea64c750f41fadbaa37b9b861558e1bfbd
split: test
type: lyon-nlp/alloprof
metrics:
- type: main_score
value: 26.772000000000002
- type: map_at_1
value: 15.543999999999999
- type: map_at_10
value: 22.753
- type: map_at_100
value: 23.907
- type: map_at_1000
value: 24.022
- type: map_at_20
value: 23.376
- type: map_at_3
value: 20.602999999999998
- type: map_at_5
value: 21.747
- type: mrr_at_1
value: 15.544041450777202
- type: mrr_at_10
value: 22.753121830194367
- type: mrr_at_100
value: 23.906950405755307
- type: mrr_at_1000
value: 24.02181335668695
- type: mrr_at_20
value: 23.375936768501013
- type: mrr_at_3
value: 20.603051237766266
- type: mrr_at_5
value: 21.747265400115143
- type: nauc_map_at_1000_diff1
value: 30.597349316847435
- type: nauc_map_at_1000_max
value: 26.64237521385283
- type: nauc_map_at_1000_std
value: -1.5619614188301716
- type: nauc_map_at_100_diff1
value: 30.562856763233114
- type: nauc_map_at_100_max
value: 26.640821876365262
- type: nauc_map_at_100_std
value: -1.5387598819000152
- type: nauc_map_at_10_diff1
value: 30.842152978574504
- type: nauc_map_at_10_max
value: 26.233032964550823
- type: nauc_map_at_10_std
value: -2.071584106514715
- type: nauc_map_at_1_diff1
value: 37.05891184379572
- type: nauc_map_at_1_max
value: 26.188171949270767
- type: nauc_map_at_1_std
value: -4.061070218304633
- type: nauc_map_at_20_diff1
value: 30.633856615001783
- type: nauc_map_at_20_max
value: 26.39938718506948
- type: nauc_map_at_20_std
value: -1.8598112266936615
- type: nauc_map_at_3_diff1
value: 31.73021739176132
- type: nauc_map_at_3_max
value: 25.95565070523295
- type: nauc_map_at_3_std
value: -2.9679328825296487
- type: nauc_map_at_5_diff1
value: 31.16505305401185
- type: nauc_map_at_5_max
value: 26.12946343604871
- type: nauc_map_at_5_std
value: -2.3315281803357846
- type: nauc_mrr_at_1000_diff1
value: 30.597349316847435
- type: nauc_mrr_at_1000_max
value: 26.64237521385283
- type: nauc_mrr_at_1000_std
value: -1.5619614188301716
- type: nauc_mrr_at_100_diff1
value: 30.562856763233114
- type: nauc_mrr_at_100_max
value: 26.640821876365262
- type: nauc_mrr_at_100_std
value: -1.5387598819000152
- type: nauc_mrr_at_10_diff1
value: 30.842152978574504
- type: nauc_mrr_at_10_max
value: 26.233032964550823
- type: nauc_mrr_at_10_std
value: -2.071584106514715
- type: nauc_mrr_at_1_diff1
value: 37.05891184379572
- type: nauc_mrr_at_1_max
value: 26.188171949270767
- type: nauc_mrr_at_1_std
value: -4.061070218304633
- type: nauc_mrr_at_20_diff1
value: 30.633856615001783
- type: nauc_mrr_at_20_max
value: 26.39938718506948
- type: nauc_mrr_at_20_std
value: -1.8598112266936615
- type: nauc_mrr_at_3_diff1
value: 31.73021739176132
- type: nauc_mrr_at_3_max
value: 25.95565070523295
- type: nauc_mrr_at_3_std
value: -2.9679328825296487
- type: nauc_mrr_at_5_diff1
value: 31.16505305401185
- type: nauc_mrr_at_5_max
value: 26.12946343604871
- type: nauc_mrr_at_5_std
value: -2.3315281803357846
- type: nauc_ndcg_at_1000_diff1
value: 28.542272012556808
- type: nauc_ndcg_at_1000_max
value: 28.099057766575363
- type: nauc_ndcg_at_1000_std
value: 0.865040955701297
- type: nauc_ndcg_at_100_diff1
value: 27.53396397771733
- type: nauc_ndcg_at_100_max
value: 28.675370109671817
- type: nauc_ndcg_at_100_std
value: 2.0073642107834413
- type: nauc_ndcg_at_10_diff1
value: 28.88580934785079
- type: nauc_ndcg_at_10_max
value: 26.52944818230396
- type: nauc_ndcg_at_10_std
value: -1.001937885275267
- type: nauc_ndcg_at_1_diff1
value: 37.05891184379572
- type: nauc_ndcg_at_1_max
value: 26.188171949270767
- type: nauc_ndcg_at_1_std
value: -4.061070218304633
- type: nauc_ndcg_at_20_diff1
value: 28.065122297591422
- type: nauc_ndcg_at_20_max
value: 27.07981760550229
- type: nauc_ndcg_at_20_std
value: -0.225359939844393
- type: nauc_ndcg_at_3_diff1
value: 30.47314525118426
- type: nauc_ndcg_at_3_max
value: 25.99046830079759
- type: nauc_ndcg_at_3_std
value: -2.6936107211011833
- type: nauc_ndcg_at_5_diff1
value: 29.54100159108883
- type: nauc_ndcg_at_5_max
value: 26.292686467196447
- type: nauc_ndcg_at_5_std
value: -1.5956457592099158
- type: nauc_precision_at_1000_diff1
value: 2.3491014548429248
- type: nauc_precision_at_1000_max
value: 59.33160809648938
- type: nauc_precision_at_1000_std
value: 50.516412095351214
- type: nauc_precision_at_100_diff1
value: 15.623313244519919
- type: nauc_precision_at_100_max
value: 38.965740747548246
- type: nauc_precision_at_100_std
value: 19.136219742706444
- type: nauc_precision_at_10_diff1
value: 24.1800008562702
- type: nauc_precision_at_10_max
value: 27.346093197617805
- type: nauc_precision_at_10_std
value: 1.778885052468617
- type: nauc_precision_at_1_diff1
value: 37.05891184379572
- type: nauc_precision_at_1_max
value: 26.188171949270767
- type: nauc_precision_at_1_std
value: -4.061070218304633
- type: nauc_precision_at_20_diff1
value: 21.09866887082253
- type: nauc_precision_at_20_max
value: 29.193940395285384
- type: nauc_precision_at_20_std
value: 4.565925881101386
- type: nauc_precision_at_3_diff1
value: 27.429462528917625
- type: nauc_precision_at_3_max
value: 26.1021029497939
- type: nauc_precision_at_3_std
value: -2.024396493639844
- type: nauc_precision_at_5_diff1
value: 25.643951914531794
- type: nauc_precision_at_5_max
value: 26.74157917546747
- type: nauc_precision_at_5_std
value: 0.27549479074549393
- type: nauc_recall_at_1000_diff1
value: 2.34910145484277
- type: nauc_recall_at_1000_max
value: 59.331608096489894
- type: nauc_recall_at_1000_std
value: 50.5164120953505
- type: nauc_recall_at_100_diff1
value: 15.623313244519943
- type: nauc_recall_at_100_max
value: 38.96574074754826
- type: nauc_recall_at_100_std
value: 19.136219742706498
- type: nauc_recall_at_10_diff1
value: 24.180000856270194
- type: nauc_recall_at_10_max
value: 27.346093197617826
- type: nauc_recall_at_10_std
value: 1.7788850524686304
- type: nauc_recall_at_1_diff1
value: 37.05891184379572
- type: nauc_recall_at_1_max
value: 26.188171949270767
- type: nauc_recall_at_1_std
value: -4.061070218304633
- type: nauc_recall_at_20_diff1
value: 21.098668870822497
- type: nauc_recall_at_20_max
value: 29.193940395285367
- type: nauc_recall_at_20_std
value: 4.565925881101381
- type: nauc_recall_at_3_diff1
value: 27.429462528917643
- type: nauc_recall_at_3_max
value: 26.102102949793938
- type: nauc_recall_at_3_std
value: -2.02439649363985
- type: nauc_recall_at_5_diff1
value: 25.643951914531783
- type: nauc_recall_at_5_max
value: 26.74157917546745
- type: nauc_recall_at_5_std
value: 0.27549479074545175
- type: ndcg_at_1
value: 15.543999999999999
- type: ndcg_at_10
value: 26.772000000000002
- type: ndcg_at_100
value: 32.975
- type: ndcg_at_1000
value: 36.281
- type: ndcg_at_20
value: 29.055999999999997
- type: ndcg_at_3
value: 22.284000000000002
- type: ndcg_at_5
value: 24.356
- type: precision_at_1
value: 15.543999999999999
- type: precision_at_10
value: 3.964
- type: precision_at_100
value: 0.7000000000000001
- type: precision_at_1000
value: 0.097
- type: precision_at_20
value: 2.435
- type: precision_at_3
value: 9.053
- type: precision_at_5
value: 6.442
- type: recall_at_1
value: 15.543999999999999
- type: recall_at_10
value: 39.637
- type: recall_at_100
value: 69.991
- type: recall_at_1000
value: 96.503
- type: recall_at_20
value: 48.705
- type: recall_at_3
value: 27.159
- type: recall_at_5
value: 32.211
task:
type: Retrieval
- dataset:
config: fr
name: MTEB AmazonReviewsClassification (fr)
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
split: test
type: mteb/amazon_reviews_multi
metrics:
- type: accuracy
value: 38.164
- type: f1
value: 37.81107303250945
- type: f1_weighted
value: 37.81107303250945
- type: main_score
value: 38.164
task:
type: Classification
- dataset:
config: default
name: MTEB BSARDRetrieval (default)
revision: 5effa1b9b5fa3b0f9e12523e6e43e5f86a6e6d59
split: test
type: maastrichtlawtech/bsard
metrics:
- type: main_score
value: 50.449999999999996
- type: map_at_1
value: 9.459
- type: map_at_10
value: 14.346999999999998
- type: map_at_100
value: 15.213
- type: map_at_1000
value: 15.312999999999999
- type: map_at_20
value: 14.841
- type: map_at_3
value: 12.763
- type: map_at_5
value: 13.619
- type: mrr_at_1
value: 9.45945945945946
- type: mrr_at_10
value: 14.347025597025597
- type: mrr_at_100
value: 15.212963730117224
- type: mrr_at_1000
value: 15.313314069254291
- type: mrr_at_20
value: 14.840973681917951
- type: mrr_at_3
value: 12.762762762762762
- type: mrr_at_5
value: 13.618618618618619
- type: nauc_map_at_1000_diff1
value: 11.8364460738485
- type: nauc_map_at_1000_max
value: 7.562827619460948
- type: nauc_map_at_1000_std
value: 2.315825902670382
- type: nauc_map_at_100_diff1
value: 11.823470732465031
- type: nauc_map_at_100_max
value: 7.570034804866984
- type: nauc_map_at_100_std
value: 2.2251049548051105
- type: nauc_map_at_10_diff1
value: 11.670870371759818
- type: nauc_map_at_10_max
value: 7.497792699176387
- type: nauc_map_at_10_std
value: 1.9741720038914088
- type: nauc_map_at_1_diff1
value: 18.88088779812427
- type: nauc_map_at_1_max
value: 1.6915776071097226
- type: nauc_map_at_1_std
value: 5.905759721184708
- type: nauc_map_at_20_diff1
value: 11.666430637618728
- type: nauc_map_at_20_max
value: 7.630110149918852
- type: nauc_map_at_20_std
value: 2.086448133894994
- type: nauc_map_at_3_diff1
value: 12.02242960274049
- type: nauc_map_at_3_max
value: 6.956922237017362
- type: nauc_map_at_3_std
value: 1.144349263014216
- type: nauc_map_at_5_diff1
value: 11.81097133293566
- type: nauc_map_at_5_max
value: 7.9581514219874485
- type: nauc_map_at_5_std
value: 2.1533332445334072
- type: nauc_mrr_at_1000_diff1
value: 11.8364460738485
- type: nauc_mrr_at_1000_max
value: 7.562827619460948
- type: nauc_mrr_at_1000_std
value: 2.315825902670382
- type: nauc_mrr_at_100_diff1
value: 11.823470732465031
- type: nauc_mrr_at_100_max
value: 7.570034804866984
- type: nauc_mrr_at_100_std
value: 2.2251049548051105
- type: nauc_mrr_at_10_diff1
value: 11.670870371759818
- type: nauc_mrr_at_10_max
value: 7.497792699176387
- type: nauc_mrr_at_10_std
value: 1.9741720038914088
- type: nauc_mrr_at_1_diff1
value: 18.88088779812427
- type: nauc_mrr_at_1_max
value: 1.6915776071097226
- type: nauc_mrr_at_1_std
value: 5.905759721184708
- type: nauc_mrr_at_20_diff1
value: 11.666430637618728
- type: nauc_mrr_at_20_max
value: 7.630110149918852
- type: nauc_mrr_at_20_std
value: 2.086448133894994
- type: nauc_mrr_at_3_diff1
value: 12.02242960274049
- type: nauc_mrr_at_3_max
value: 6.956922237017362
- type: nauc_mrr_at_3_std
value: 1.144349263014216
- type: nauc_mrr_at_5_diff1
value: 11.81097133293566
- type: nauc_mrr_at_5_max
value: 7.9581514219874485
- type: nauc_mrr_at_5_std
value: 2.1533332445334072
- type: nauc_ndcg_at_1000_diff1
value: 10.484731306768154
- type: nauc_ndcg_at_1000_max
value: 8.753073859813403
- type: nauc_ndcg_at_1000_std
value: 4.587003936388711
- type: nauc_ndcg_at_100_diff1
value: 9.848875670873582
- type: nauc_ndcg_at_100_max
value: 8.25534458391377
- type: nauc_ndcg_at_100_std
value: 2.4414259554432753
- type: nauc_ndcg_at_10_diff1
value: 9.206402248396946
- type: nauc_ndcg_at_10_max
value: 8.5927885053332
- type: nauc_ndcg_at_10_std
value: 1.1370639724407328
- type: nauc_ndcg_at_1_diff1
value: 18.88088779812427
- type: nauc_ndcg_at_1_max
value: 1.6915776071097226
- type: nauc_ndcg_at_1_std
value: 5.905759721184708
- type: nauc_ndcg_at_20_diff1
value: 9.632954524825056
- type: nauc_ndcg_at_20_max
value: 9.263814904284732
- type: nauc_ndcg_at_20_std
value: 1.7848394563063672
- type: nauc_ndcg_at_3_diff1
value: 9.794897172936222
- type: nauc_ndcg_at_3_max
value: 7.8110311191727355
- type: nauc_ndcg_at_3_std
value: -0.39574204179853867
- type: nauc_ndcg_at_5_diff1
value: 9.471323179357626
- type: nauc_ndcg_at_5_max
value: 9.465523343334816
- type: nauc_ndcg_at_5_std
value: 1.4149266752456098
- type: nauc_precision_at_1000_diff1
value: 10.166075851342518
- type: nauc_precision_at_1000_max
value: 14.066021499855424
- type: nauc_precision_at_1000_std
value: 24.379900864639076
- type: nauc_precision_at_100_diff1
value: 5.942604500628994
- type: nauc_precision_at_100_max
value: 8.08635556229695
- type: nauc_precision_at_100_std
value: 4.216874709016507
- type: nauc_precision_at_10_diff1
value: 4.149102926209563
- type: nauc_precision_at_10_max
value: 10.695497632873838
- type: nauc_precision_at_10_std
value: -0.36788079476155144
- type: nauc_precision_at_1_diff1
value: 18.88088779812427
- type: nauc_precision_at_1_max
value: 1.6915776071097226
- type: nauc_precision_at_1_std
value: 5.905759721184708
- type: nauc_precision_at_20_diff1
value: 6.290935314825552
- type: nauc_precision_at_20_max
value: 12.853306112504118
- type: nauc_precision_at_20_std
value: 1.9752438622164594
- type: nauc_precision_at_3_diff1
value: 4.614084654371029
- type: nauc_precision_at_3_max
value: 9.631615263215073
- type: nauc_precision_at_3_std
value: -3.9764680082834536
- type: nauc_precision_at_5_diff1
value: 4.370931213610563
- type: nauc_precision_at_5_max
value: 12.716210094112004
- type: nauc_precision_at_5_std
value: 0.009730736100650395
- type: nauc_recall_at_1000_diff1
value: 10.166075851342567
- type: nauc_recall_at_1000_max
value: 14.066021499855522
- type: nauc_recall_at_1000_std
value: 24.379900864639094
- type: nauc_recall_at_100_diff1
value: 5.9426045006289865
- type: nauc_recall_at_100_max
value: 8.086355562296912
- type: nauc_recall_at_100_std
value: 4.216874709016509
- type: nauc_recall_at_10_diff1
value: 4.149102926209514
- type: nauc_recall_at_10_max
value: 10.695497632873794
- type: nauc_recall_at_10_std
value: -0.36788079476160407
- type: nauc_recall_at_1_diff1
value: 18.88088779812427
- type: nauc_recall_at_1_max
value: 1.6915776071097226
- type: nauc_recall_at_1_std
value: 5.905759721184708
- type: nauc_recall_at_20_diff1
value: 6.290935314825502
- type: nauc_recall_at_20_max
value: 12.853306112504068
- type: nauc_recall_at_20_std
value: 1.9752438622164061
- type: nauc_recall_at_3_diff1
value: 4.61408465437105
- type: nauc_recall_at_3_max
value: 9.631615263215096
- type: nauc_recall_at_3_std
value: -3.9764680082834523
- type: nauc_recall_at_5_diff1
value: 4.370931213610528
- type: nauc_recall_at_5_max
value: 12.716210094111954
- type: nauc_recall_at_5_std
value: 0.009730736100608914
- type: ndcg_at_1
value: 9.459
- type: ndcg_at_10
value: 17.255000000000003
- type: ndcg_at_100
value: 22.063
- type: ndcg_at_1000
value: 25.433
- type: ndcg_at_20
value: 19.144
- type: ndcg_at_3
value: 13.985
- type: ndcg_at_5
value: 15.498000000000001
- type: precision_at_1
value: 9.459
- type: precision_at_10
value: 2.658
- type: precision_at_100
value: 0.505
- type: precision_at_1000
value: 0.078
- type: precision_at_20
value: 1.712
- type: precision_at_3
value: 5.856
- type: precision_at_5
value: 4.234
- type: recall_at_1
value: 9.459
- type: recall_at_10
value: 26.577
- type: recall_at_100
value: 50.449999999999996
- type: recall_at_1000
value: 78.378
- type: recall_at_20
value: 34.233999999999995
- type: recall_at_3
value: 17.568
- type: recall_at_5
value: 21.171
task:
type: Retrieval
- dataset:
config: default
name: MTEB HALClusteringS2S (default)
revision: e06ebbbb123f8144bef1a5d18796f3dec9ae2915
split: test
type: lyon-nlp/clustering-hal-s2s
metrics:
- type: main_score
value: 22.166617958817174
- type: v_measure
value: 22.166617958817174
- type: v_measure_std
value: 3.0337927223518952
task:
type: Clustering
- dataset:
config: fr
name: MTEB MLSUMClusteringP2P (fr)
revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7
split: test
type: reciTAL/mlsum
metrics:
- type: main_score
value: 43.477227787877865
- type: v_measure
value: 43.477227787877865
- type: v_measure_std
value: 2.1410189575891767
task:
type: Clustering
- dataset:
config: fr
name: MTEB MLSUMClusteringS2S (fr)
revision: b5d54f8f3b61ae17845046286940f03c6bc79bc7
split: test
type: reciTAL/mlsum
metrics:
- type: main_score
value: 43.49160373246708
- type: v_measure
value: 43.49160373246708
- type: v_measure_std
value: 2.447685044393075
task:
type: Clustering
- dataset:
config: fr
name: MTEB MTOPDomainClassification (fr)
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
split: test
type: mteb/mtop_domain
metrics:
- type: accuracy
value: 80.05950516755401
- type: f1
value: 80.64260140711778
- type: f1_weighted
value: 80.05975231330979
- type: main_score
value: 80.05950516755401
task:
type: Classification
- dataset:
config: fr
name: MTEB MTOPIntentClassification (fr)
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
split: test
type: mteb/mtop_intent
metrics:
- type: accuracy
value: 57.41622298778578
- type: f1
value: 40.811068656490676
- type: f1_weighted
value: 61.38872619156185
- type: main_score
value: 57.41622298778578
task:
type: Classification
- dataset:
config: fra
name: MTEB MasakhaNEWSClassification (fra)
revision: 18193f187b92da67168c655c9973a165ed9593dd
split: test
type: mteb/masakhanews
metrics:
- type: accuracy
value: 72.6303317535545
- type: f1
value: 69.09603253255243
- type: f1_weighted
value: 73.07798963505878
- type: main_score
value: 72.6303317535545
task:
type: Classification
- dataset:
config: fra
name: MTEB MasakhaNEWSClusteringP2P (fra)
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
split: test
type: masakhane/masakhanews
metrics:
- type: main_score
value: 47.50151745798914
- type: v_measure
value: 47.50151745798914
- type: v_measure_std
value: 31.28071132108758
task:
type: Clustering
- dataset:
config: fra
name: MTEB MasakhaNEWSClusteringS2S (fra)
revision: 8ccc72e69e65f40c70e117d8b3c08306bb788b60
split: test
type: masakhane/masakhanews
metrics:
- type: main_score
value: 42.314391162132
- type: v_measure
value: 42.314391162132
- type: v_measure_std
value: 33.322476978491345
task:
type: Clustering
- dataset:
config: fr
name: MTEB MassiveIntentClassification (fr)
revision: 4672e20407010da34463acc759c162ca9734bca6
split: test
type: mteb/amazon_massive_intent
metrics:
- type: accuracy
value: 61.398789509078675
- type: f1
value: 59.27069381204525
- type: f1_weighted
value: 61.703579039124314
- type: main_score
value: 61.398789509078675
task:
type: Classification
- dataset:
config: fr
name: MTEB MassiveScenarioClassification (fr)
revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
split: test
type: mteb/amazon_massive_scenario
metrics:
- type: accuracy
value: 68.38601210490921
- type: f1
value: 68.6191510801706
- type: f1_weighted
value: 68.4297764272372
- type: main_score
value: 68.38601210490921
task:
type: Classification
- dataset:
config: fr
name: MTEB MintakaRetrieval (fr)
revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e
split: test
type: jinaai/mintakaqa
metrics:
- type: main_score
value: 16.854
- type: map_at_1
value: 9.132
- type: map_at_10
value: 13.957
- type: map_at_100
value: 14.831
- type: map_at_1000
value: 14.97
- type: map_at_20
value: 14.411999999999999
- type: map_at_3
value: 12.34
- type: map_at_5
value: 13.139999999999999
- type: mrr_at_1
value: 9.131859131859132
- type: mrr_at_10
value: 13.957301457301458
- type: mrr_at_100
value: 14.831480640740228
- type: mrr_at_1000
value: 14.970082403750823
- type: mrr_at_20
value: 14.411634316644168
- type: mrr_at_3
value: 12.339612339612339
- type: mrr_at_5
value: 13.140185640185639
- type: nauc_map_at_1000_diff1
value: 13.869933549977034
- type: nauc_map_at_1000_max
value: 23.573005148821352
- type: nauc_map_at_1000_std
value: 19.000589731201057
- type: nauc_map_at_100_diff1
value: 13.827500860067222
- type: nauc_map_at_100_max
value: 23.546517242016314
- type: nauc_map_at_100_std
value: 19.033201633415356
- type: nauc_map_at_10_diff1
value: 14.359042524386338
- type: nauc_map_at_10_max
value: 24.26886754737575
- type: nauc_map_at_10_std
value: 18.979267787610873
- type: nauc_map_at_1_diff1
value: 24.427979037138254
- type: nauc_map_at_1_max
value: 24.932017493976094
- type: nauc_map_at_1_std
value: 15.26890166620689
- type: nauc_map_at_20_diff1
value: 13.977303066154617
- type: nauc_map_at_20_max
value: 23.825551540802962
- type: nauc_map_at_20_std
value: 19.134945526049805
- type: nauc_map_at_3_diff1
value: 16.489213644088174
- type: nauc_map_at_3_max
value: 24.83455814378989
- type: nauc_map_at_3_std
value: 17.791525469033704
- type: nauc_map_at_5_diff1
value: 15.399163330752305
- type: nauc_map_at_5_max
value: 24.37862829256271
- type: nauc_map_at_5_std
value: 18.333714194446728
- type: nauc_mrr_at_1000_diff1
value: 13.869933549977034
- type: nauc_mrr_at_1000_max
value: 23.573005148821352
- type: nauc_mrr_at_1000_std
value: 19.000589731201057
- type: nauc_mrr_at_100_diff1
value: 13.827500860067222
- type: nauc_mrr_at_100_max
value: 23.546517242016314
- type: nauc_mrr_at_100_std
value: 19.033201633415356
- type: nauc_mrr_at_10_diff1
value: 14.359042524386338
- type: nauc_mrr_at_10_max
value: 24.26886754737575
- type: nauc_mrr_at_10_std
value: 18.979267787610873
- type: nauc_mrr_at_1_diff1
value: 24.427979037138254
- type: nauc_mrr_at_1_max
value: 24.932017493976094
- type: nauc_mrr_at_1_std
value: 15.26890166620689
- type: nauc_mrr_at_20_diff1
value: 13.977303066154617
- type: nauc_mrr_at_20_max
value: 23.825551540802962
- type: nauc_mrr_at_20_std
value: 19.134945526049805
- type: nauc_mrr_at_3_diff1
value: 16.489213644088174
- type: nauc_mrr_at_3_max
value: 24.83455814378989
- type: nauc_mrr_at_3_std
value: 17.791525469033704
- type: nauc_mrr_at_5_diff1
value: 15.399163330752305
- type: nauc_mrr_at_5_max
value: 24.37862829256271
- type: nauc_mrr_at_5_std
value: 18.333714194446728
- type: nauc_ndcg_at_1000_diff1
value: 10.064604490374695
- type: nauc_ndcg_at_1000_max
value: 22.103937884193908
- type: nauc_ndcg_at_1000_std
value: 20.06975661685462
- type: nauc_ndcg_at_100_diff1
value: 9.023906499604244
- type: nauc_ndcg_at_100_max
value: 21.267338704691173
- type: nauc_ndcg_at_100_std
value: 20.906031366344678
- type: nauc_ndcg_at_10_diff1
value: 10.698599153498867
- type: nauc_ndcg_at_10_max
value: 24.09091983922221
- type: nauc_ndcg_at_10_std
value: 20.80728749400035
- type: nauc_ndcg_at_1_diff1
value: 24.427979037138254
- type: nauc_ndcg_at_1_max
value: 24.932017493976094
- type: nauc_ndcg_at_1_std
value: 15.26890166620689
- type: nauc_ndcg_at_20_diff1
value: 9.678851886824095
- type: nauc_ndcg_at_20_max
value: 22.769227734404144
- type: nauc_ndcg_at_20_std
value: 21.302337978625847
- type: nauc_ndcg_at_3_diff1
value: 14.38237669131778
- type: nauc_ndcg_at_3_max
value: 25.035309831892082
- type: nauc_ndcg_at_3_std
value: 18.61132699179348
- type: nauc_ndcg_at_5_diff1
value: 12.805144383213213
- type: nauc_ndcg_at_5_max
value: 24.311996297167674
- type: nauc_ndcg_at_5_std
value: 19.45989756442976
- type: nauc_precision_at_1000_diff1
value: -6.501006177479093
- type: nauc_precision_at_1000_max
value: 15.233639914960836
- type: nauc_precision_at_1000_std
value: 19.69510141955613
- type: nauc_precision_at_100_diff1
value: -1.2343292697815695
- type: nauc_precision_at_100_max
value: 14.821492227318918
- type: nauc_precision_at_100_std
value: 24.815010132148707
- type: nauc_precision_at_10_diff1
value: 3.063443341827621
- type: nauc_precision_at_10_max
value: 23.703724350410138
- type: nauc_precision_at_10_std
value: 24.76419500191453
- type: nauc_precision_at_1_diff1
value: 24.427979037138254
- type: nauc_precision_at_1_max
value: 24.932017493976094
- type: nauc_precision_at_1_std
value: 15.26890166620689
- type: nauc_precision_at_20_diff1
value: 0.9187523699274829
- type: nauc_precision_at_20_max
value: 20.17033760734205
- type: nauc_precision_at_20_std
value: 25.917732068306144
- type: nauc_precision_at_3_diff1
value: 9.480920817019808
- type: nauc_precision_at_3_max
value: 25.556658656709114
- type: nauc_precision_at_3_std
value: 20.55458409998131
- type: nauc_precision_at_5_diff1
value: 7.259255566252315
- type: nauc_precision_at_5_max
value: 24.1716555525607
- type: nauc_precision_at_5_std
value: 21.9376099722901
- type: nauc_recall_at_1000_diff1
value: -6.501006177478934
- type: nauc_recall_at_1000_max
value: 15.23363991496086
- type: nauc_recall_at_1000_std
value: 19.695101419555975
- type: nauc_recall_at_100_diff1
value: -1.2343292697815282
- type: nauc_recall_at_100_max
value: 14.82149222731895
- type: nauc_recall_at_100_std
value: 24.815010132148736
- type: nauc_recall_at_10_diff1
value: 3.0634433418276243
- type: nauc_recall_at_10_max
value: 23.703724350410134
- type: nauc_recall_at_10_std
value: 24.764195001914526
- type: nauc_recall_at_1_diff1
value: 24.427979037138254
- type: nauc_recall_at_1_max
value: 24.932017493976094
- type: nauc_recall_at_1_std
value: 15.26890166620689
- type: nauc_recall_at_20_diff1
value: 0.918752369927499
- type: nauc_recall_at_20_max
value: 20.17033760734205
- type: nauc_recall_at_20_std
value: 25.917732068306133
- type: nauc_recall_at_3_diff1
value: 9.48092081701983
- type: nauc_recall_at_3_max
value: 25.556658656709125
- type: nauc_recall_at_3_std
value: 20.55458409998131
- type: nauc_recall_at_5_diff1
value: 7.259255566252292
- type: nauc_recall_at_5_max
value: 24.1716555525607
- type: nauc_recall_at_5_std
value: 21.937609972290094
- type: ndcg_at_1
value: 9.132
- type: ndcg_at_10
value: 16.854
- type: ndcg_at_100
value: 21.715999999999998
- type: ndcg_at_1000
value: 26.423999999999996
- type: ndcg_at_20
value: 18.496000000000002
- type: ndcg_at_3
value: 13.407
- type: ndcg_at_5
value: 14.863000000000001
- type: precision_at_1
value: 9.132
- type: precision_at_10
value: 2.625
- type: precision_at_100
value: 0.505
- type: precision_at_1000
value: 0.09
- type: precision_at_20
value: 1.636
- type: precision_at_3
value: 5.501
- type: precision_at_5
value: 4.013
- type: recall_at_1
value: 9.132
- type: recall_at_10
value: 26.249
- type: recall_at_100
value: 50.449999999999996
- type: recall_at_1000
value: 89.72200000000001
- type: recall_at_20
value: 32.719
- type: recall_at_3
value: 16.503
- type: recall_at_5
value: 20.066
task:
type: Retrieval
- dataset:
config: fr
name: MTEB OpusparcusPC (fr)
revision: 9e9b1f8ef51616073f47f306f7f47dd91663f86a
split: test
type: GEM/opusparcus
metrics:
- type: main_score
value: 100
task:
type: PairClassification
- dataset:
config: fr
name: MTEB PawsX (fr)
revision: 8a04d940a42cd40658986fdd8e3da561533a3646
split: test
type: google-research-datasets/paws-x
metrics:
- type: main_score
value: 57.47426331832951
task:
type: PairClassification
- dataset:
config: default
name: MTEB SICKFr (default)
revision: e077ab4cf4774a1e36d86d593b150422fafd8e8a
split: test
type: Lajavaness/SICK-fr
metrics:
- type: cosine_pearson
value: 87.16612243314145
- type: cosine_spearman
value: 82.48638006860072
- type: euclidean_pearson
value: 84.82287788378513
- type: euclidean_spearman
value: 82.43575328556403
- type: main_score
value: 82.48638006860072
- type: manhattan_pearson
value: 85.19582681908197
- type: manhattan_spearman
value: 82.54536594850015
task:
type: STS
- dataset:
config: fr
name: MTEB STS22 (fr)
revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3
split: test
type: mteb/sts22-crosslingual-sts
metrics:
- type: cosine_pearson
value: 73.76992335121398
- type: cosine_spearman
value: 75.17870675221324
- type: euclidean_pearson
value: 68.02453778209556
- type: euclidean_spearman
value: 75.22743994750913
- type: main_score
value: 75.17870675221324
- type: manhattan_pearson
value: 73.78873447472732
- type: manhattan_spearman
value: 75.53146863179809
task:
type: STS
- dataset:
config: fr
name: MTEB STSBenchmarkMultilingualSTS (fr)
revision: 29afa2569dcedaaa2fe6a3dcfebab33d28b82e8c
split: test
type: mteb/stsb_multi_mt
metrics:
- type: cosine_pearson
value: 82.09583670528698
- type: cosine_spearman
value: 82.39603745417014
- type: euclidean_pearson
value: 82.8742665513042
- type: euclidean_spearman
value: 83.03871235986725
- type: main_score
value: 82.39603745417014
- type: manhattan_pearson
value: 83.18707033279978
- type: manhattan_spearman
value: 83.17121779758206
task:
type: STS
- dataset:
config: default
name: MTEB SummEvalFr (default)
revision: b385812de6a9577b6f4d0f88c6a6e35395a94054
split: test
type: lyon-nlp/summarization-summeval-fr-p2p
metrics:
- type: cosine_pearson
value: 28.811168826829213
- type: cosine_spearman
value: 30.705809960659035
- type: dot_pearson
value: 28.109054287846774
- type: dot_spearman
value: 29.718047453954743
- type: main_score
value: 30.705809960659035
- type: pearson
value: 28.811169012827115
- type: spearman
value: 30.705809960659035
task:
type: Summarization
- dataset:
config: default
name: MTEB SyntecReranking (default)
revision: daf0863838cd9e3ba50544cdce3ac2b338a1b0ad
split: test
type: lyon-nlp/mteb-fr-reranking-syntec-s2p
metrics:
- type: main_score
value: 77.06111111111112
- type: map
value: 77.06111111111112
- type: mrr
value: 77.06111111111112
- type: nAUC_map_diff1
value: 54.75959885778961
- type: nAUC_map_max
value: 5.404205440070597
- type: nAUC_map_std
value: 14.368157545809618
- type: nAUC_mrr_diff1
value: 54.75959885778961
- type: nAUC_mrr_max
value: 5.404205440070597
- type: nAUC_mrr_std
value: 14.368157545809618
task:
type: Reranking
- dataset:
config: default
name: MTEB SyntecRetrieval (default)
revision: 19661ccdca4dfc2d15122d776b61685f48c68ca9
split: test
type: lyon-nlp/mteb-fr-retrieval-syntec-s2p
metrics:
- type: main_score
value: 66.591
- type: map_at_1
value: 46
- type: map_at_10
value: 59.509
- type: map_at_100
value: 60.144
- type: map_at_1000
value: 60.144
- type: map_at_20
value: 59.94199999999999
- type: map_at_3
value: 56.00000000000001
- type: map_at_5
value: 57.8
- type: mrr_at_1
value: 46
- type: mrr_at_10
value: 59.50912698412698
- type: mrr_at_100
value: 60.143565584742056
- type: mrr_at_1000
value: 60.143565584742056
- type: mrr_at_20
value: 59.94179235796883
- type: mrr_at_3
value: 56.00000000000001
- type: mrr_at_5
value: 57.8
- type: nauc_map_at_1000_diff1
value: 58.42057638169377
- type: nauc_map_at_1000_max
value: 28.870632165878934
- type: nauc_map_at_1000_std
value: -11.910078246389578
- type: nauc_map_at_100_diff1
value: 58.42057638169377
- type: nauc_map_at_100_max
value: 28.870632165878934
- type: nauc_map_at_100_std
value: -11.910078246389578
- type: nauc_map_at_10_diff1
value: 58.022067328717156
- type: nauc_map_at_10_max
value: 28.74614532609343
- type: nauc_map_at_10_std
value: -11.575101736987184
- type: nauc_map_at_1_diff1
value: 59.75927045281334
- type: nauc_map_at_1_max
value: 28.6773015163859
- type: nauc_map_at_1_std
value: -12.547633439331438
- type: nauc_map_at_20_diff1
value: 58.418048056669136
- type: nauc_map_at_20_max
value: 29.129379094542745
- type: nauc_map_at_20_std
value: -11.368609008653982
- type: nauc_map_at_3_diff1
value: 58.55896157230859
- type: nauc_map_at_3_max
value: 27.707538867703107
- type: nauc_map_at_3_std
value: -13.95448323066401
- type: nauc_map_at_5_diff1
value: 57.375613495527254
- type: nauc_map_at_5_max
value: 29.292025398027665
- type: nauc_map_at_5_std
value: -11.953987708201609
- type: nauc_mrr_at_1000_diff1
value: 58.42057638169377
- type: nauc_mrr_at_1000_max
value: 28.870632165878934
- type: nauc_mrr_at_1000_std
value: -11.910078246389578
- type: nauc_mrr_at_100_diff1
value: 58.42057638169377
- type: nauc_mrr_at_100_max
value: 28.870632165878934
- type: nauc_mrr_at_100_std
value: -11.910078246389578
- type: nauc_mrr_at_10_diff1
value: 58.022067328717156
- type: nauc_mrr_at_10_max
value: 28.74614532609343
- type: nauc_mrr_at_10_std
value: -11.575101736987184
- type: nauc_mrr_at_1_diff1
value: 59.75927045281334
- type: nauc_mrr_at_1_max
value: 28.6773015163859
- type: nauc_mrr_at_1_std
value: -12.547633439331438
- type: nauc_mrr_at_20_diff1
value: 58.418048056669136
- type: nauc_mrr_at_20_max
value: 29.129379094542745
- type: nauc_mrr_at_20_std
value: -11.368609008653982
- type: nauc_mrr_at_3_diff1
value: 58.55896157230859
- type: nauc_mrr_at_3_max
value: 27.707538867703107
- type: nauc_mrr_at_3_std
value: -13.95448323066401
- type: nauc_mrr_at_5_diff1
value: 57.375613495527254
- type: nauc_mrr_at_5_max
value: 29.292025398027665
- type: nauc_mrr_at_5_std
value: -11.953987708201609
- type: nauc_ndcg_at_1000_diff1
value: 58.01457592598891
- type: nauc_ndcg_at_1000_max
value: 28.948558605353607
- type: nauc_ndcg_at_1000_std
value: -11.425173234366834
- type: nauc_ndcg_at_100_diff1
value: 58.01457592598891
- type: nauc_ndcg_at_100_max
value: 28.948558605353607
- type: nauc_ndcg_at_100_std
value: -11.425173234366834
- type: nauc_ndcg_at_10_diff1
value: 56.205518242381615
- type: nauc_ndcg_at_10_max
value: 28.69512752017703
- type: nauc_ndcg_at_10_std
value: -9.184789755220011
- type: nauc_ndcg_at_1_diff1
value: 59.75927045281334
- type: nauc_ndcg_at_1_max
value: 28.6773015163859
- type: nauc_ndcg_at_1_std
value: -12.547633439331438
- type: nauc_ndcg_at_20_diff1
value: 57.91356226826028
- type: nauc_ndcg_at_20_max
value: 30.71076232473184
- type: nauc_ndcg_at_20_std
value: -7.83147422641797
- type: nauc_ndcg_at_3_diff1
value: 57.51631639939359
- type: nauc_ndcg_at_3_max
value: 26.733246230873103
- type: nauc_ndcg_at_3_std
value: -14.993737400509197
- type: nauc_ndcg_at_5_diff1
value: 55.07568223208887
- type: nauc_ndcg_at_5_max
value: 29.96627645637971
- type: nauc_ndcg_at_5_std
value: -10.875450985492236
- type: nauc_precision_at_1000_diff1
value: .nan
- type: nauc_precision_at_1000_max
value: .nan
- type: nauc_precision_at_1000_std
value: .nan
- type: nauc_precision_at_100_diff1
value: .nan
- type: nauc_precision_at_100_max
value: .nan
- type: nauc_precision_at_100_std
value: .nan
- type: nauc_precision_at_10_diff1
value: 40.847006270939076
- type: nauc_precision_at_10_max
value: 28.50270595309666
- type: nauc_precision_at_10_std
value: 13.84331243020359
- type: nauc_precision_at_1_diff1
value: 59.75927045281334
- type: nauc_precision_at_1_max
value: 28.6773015163859
- type: nauc_precision_at_1_std
value: -12.547633439331438
- type: nauc_precision_at_20_diff1
value: 54.81792717086857
- type: nauc_precision_at_20_max
value: 68.40336134453831
- type: nauc_precision_at_20_std
value: 68.40336134453831
- type: nauc_precision_at_3_diff1
value: 53.680088893592284
- type: nauc_precision_at_3_max
value: 23.128207841684826
- type: nauc_precision_at_3_std
value: -18.789353934070476
- type: nauc_precision_at_5_diff1
value: 44.511089724939026
- type: nauc_precision_at_5_max
value: 33.42360287370472
- type: nauc_precision_at_5_std
value: -5.402743895615482
- type: nauc_recall_at_1000_diff1
value: .nan
- type: nauc_recall_at_1000_max
value: .nan
- type: nauc_recall_at_1000_std
value: .nan
- type: nauc_recall_at_100_diff1
value: .nan
- type: nauc_recall_at_100_max
value: .nan
- type: nauc_recall_at_100_std
value: .nan
- type: nauc_recall_at_10_diff1
value: 40.847006270938934
- type: nauc_recall_at_10_max
value: 28.50270595309693
- type: nauc_recall_at_10_std
value: 13.843312430203541
- type: nauc_recall_at_1_diff1
value: 59.75927045281334
- type: nauc_recall_at_1_max
value: 28.6773015163859
- type: nauc_recall_at_1_std
value: -12.547633439331438
- type: nauc_recall_at_20_diff1
value: 54.81792717086827
- type: nauc_recall_at_20_max
value: 68.40336134453781
- type: nauc_recall_at_20_std
value: 68.40336134453781
- type: nauc_recall_at_3_diff1
value: 53.680088893592256
- type: nauc_recall_at_3_max
value: 23.128207841684798
- type: nauc_recall_at_3_std
value: -18.789353934070594
- type: nauc_recall_at_5_diff1
value: 44.51108972493898
- type: nauc_recall_at_5_max
value: 33.4236028737047
- type: nauc_recall_at_5_std
value: -5.402743895615586
- type: ndcg_at_1
value: 46
- type: ndcg_at_10
value: 66.591
- type: ndcg_at_100
value: 69.194
- type: ndcg_at_1000
value: 69.194
- type: ndcg_at_20
value: 68.12899999999999
- type: ndcg_at_3
value: 59.333000000000006
- type: ndcg_at_5
value: 62.602999999999994
- type: precision_at_1
value: 46
- type: precision_at_10
value: 8.9
- type: precision_at_100
value: 1
- type: precision_at_1000
value: 0.1
- type: precision_at_20
value: 4.75
- type: precision_at_3
value: 23
- type: precision_at_5
value: 15.4
- type: recall_at_1
value: 46
- type: recall_at_10
value: 89
- type: recall_at_100
value: 100
- type: recall_at_1000
value: 100
- type: recall_at_20
value: 95
- type: recall_at_3
value: 69
- type: recall_at_5
value: 77
task:
type: Retrieval
- dataset:
config: fra-fra
name: MTEB XPQARetrieval (fra-fra)
revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f
split: test
type: jinaai/xpqa
metrics:
- type: main_score
value: 51.273
- type: map_at_1
value: 29.424
- type: map_at_10
value: 45.402
- type: map_at_100
value: 47.093
- type: map_at_1000
value: 47.205999999999996
- type: map_at_20
value: 46.385
- type: map_at_3
value: 40.53
- type: map_at_5
value: 43.43
- type: mrr_at_1
value: 46.5954606141522
- type: mrr_at_10
value: 53.36983703138576
- type: mrr_at_100
value: 54.20150645088333
- type: mrr_at_1000
value: 54.240710261333604
- type: mrr_at_20
value: 53.91545671500989
- type: mrr_at_3
value: 51.3351134846462
- type: mrr_at_5
value: 52.38985313751668
- type: nauc_map_at_1000_diff1
value: 45.59314155477728
- type: nauc_map_at_1000_max
value: 42.5672492443023
- type: nauc_map_at_1000_std
value: 1.6285231522145658
- type: nauc_map_at_100_diff1
value: 45.55082194560616
- type: nauc_map_at_100_max
value: 42.53424640856884
- type: nauc_map_at_100_std
value: 1.602887139264404
- type: nauc_map_at_10_diff1
value: 45.84899929696291
- type: nauc_map_at_10_max
value: 41.620322180163235
- type: nauc_map_at_10_std
value: 0.8069453354454892
- type: nauc_map_at_1_diff1
value: 52.771811958456404
- type: nauc_map_at_1_max
value: 29.761953273353463
- type: nauc_map_at_1_std
value: -3.3283748548746397
- type: nauc_map_at_20_diff1
value: 45.58722684105465
- type: nauc_map_at_20_max
value: 42.23726663715404
- type: nauc_map_at_20_std
value: 1.208739135781583
- type: nauc_map_at_3_diff1
value: 47.164797479789634
- type: nauc_map_at_3_max
value: 37.74810923998812
- type: nauc_map_at_3_std
value: -1.2101464202972105
- type: nauc_map_at_5_diff1
value: 46.00944429857829
- type: nauc_map_at_5_max
value: 40.52537334442166
- type: nauc_map_at_5_std
value: 0.34989150074875564
- type: nauc_mrr_at_1000_diff1
value: 50.37662237533227
- type: nauc_mrr_at_1000_max
value: 48.04811000238343
- type: nauc_mrr_at_1000_std
value: 3.3612013483424095
- type: nauc_mrr_at_100_diff1
value: 50.369041703846676
- type: nauc_mrr_at_100_max
value: 48.07304150167221
- type: nauc_mrr_at_100_std
value: 3.4020871497842995
- type: nauc_mrr_at_10_diff1
value: 50.394047504172846
- type: nauc_mrr_at_10_max
value: 47.92217523267111
- type: nauc_mrr_at_10_std
value: 3.256671742517228
- type: nauc_mrr_at_1_diff1
value: 52.98217881313075
- type: nauc_mrr_at_1_max
value: 49.12587561275278
- type: nauc_mrr_at_1_std
value: 1.898788348978136
- type: nauc_mrr_at_20_diff1
value: 50.275293114110966
- type: nauc_mrr_at_20_max
value: 47.92794344300191
- type: nauc_mrr_at_20_std
value: 3.2373129185754057
- type: nauc_mrr_at_3_diff1
value: 51.1047993924964
- type: nauc_mrr_at_3_max
value: 47.493587547123425
- type: nauc_mrr_at_3_std
value: 1.636690244729369
- type: nauc_mrr_at_5_diff1
value: 50.74332004631521
- type: nauc_mrr_at_5_max
value: 47.84066820359849
- type: nauc_mrr_at_5_std
value: 2.7226449286978167
- type: nauc_ndcg_at_1000_diff1
value: 46.08752338112771
- type: nauc_ndcg_at_1000_max
value: 45.8391276611751
- type: nauc_ndcg_at_1000_std
value: 4.8126343332631905
- type: nauc_ndcg_at_100_diff1
value: 45.354006095294004
- type: nauc_ndcg_at_100_max
value: 45.885397887754145
- type: nauc_ndcg_at_100_std
value: 5.569688679641098
- type: nauc_ndcg_at_10_diff1
value: 45.83859912037679
- type: nauc_ndcg_at_10_max
value: 43.29587921610461
- type: nauc_ndcg_at_10_std
value: 2.43414805980802
- type: nauc_ndcg_at_1_diff1
value: 52.98217881313075
- type: nauc_ndcg_at_1_max
value: 49.12587561275278
- type: nauc_ndcg_at_1_std
value: 1.898788348978136
- type: nauc_ndcg_at_20_diff1
value: 45.020369105553016
- type: nauc_ndcg_at_20_max
value: 44.21157919298904
- type: nauc_ndcg_at_20_std
value: 3.0669566546015403
- type: nauc_ndcg_at_3_diff1
value: 46.27314368474881
- type: nauc_ndcg_at_3_max
value: 42.48300233354645
- type: nauc_ndcg_at_3_std
value: 0.12637599662001878
- type: nauc_ndcg_at_5_diff1
value: 46.21294374358085
- type: nauc_ndcg_at_5_max
value: 42.2347549649749
- type: nauc_ndcg_at_5_std
value: 1.485041773198402
- type: nauc_precision_at_1000_diff1
value: -10.065802240600465
- type: nauc_precision_at_1000_max
value: 21.646326923742205
- type: nauc_precision_at_1000_std
value: 13.795418925755143
- type: nauc_precision_at_100_diff1
value: -3.5621073123774516
- type: nauc_precision_at_100_max
value: 30.75617942056576
- type: nauc_precision_at_100_std
value: 16.964896226898812
- type: nauc_precision_at_10_diff1
value: 9.80875507684663
- type: nauc_precision_at_10_max
value: 36.42717418778496
- type: nauc_precision_at_10_std
value: 7.9668245359880165
- type: nauc_precision_at_1_diff1
value: 52.98217881313075
- type: nauc_precision_at_1_max
value: 49.12587561275278
- type: nauc_precision_at_1_std
value: 1.898788348978136
- type: nauc_precision_at_20_diff1
value: 3.775640202932283
- type: nauc_precision_at_20_max
value: 34.78244550057632
- type: nauc_precision_at_20_std
value: 10.477583248238298
- type: nauc_precision_at_3_diff1
value: 20.840731694529214
- type: nauc_precision_at_3_max
value: 39.361599659000504
- type: nauc_precision_at_3_std
value: 3.541830869048572
- type: nauc_precision_at_5_diff1
value: 14.780546251243432
- type: nauc_precision_at_5_max
value: 39.35968046331222
- type: nauc_precision_at_5_std
value: 7.008389769740811
- type: nauc_recall_at_1000_diff1
value: 40.53889865128061
- type: nauc_recall_at_1000_max
value: 69.77625252525398
- type: nauc_recall_at_1000_std
value: 67.25856046695323
- type: nauc_recall_at_100_diff1
value: 34.27269500934244
- type: nauc_recall_at_100_max
value: 49.52347817993674
- type: nauc_recall_at_100_std
value: 26.779741910265354
- type: nauc_recall_at_10_diff1
value: 39.75761979107115
- type: nauc_recall_at_10_max
value: 38.18917028855593
- type: nauc_recall_at_10_std
value: 4.61011336014113
- type: nauc_recall_at_1_diff1
value: 52.771811958456404
- type: nauc_recall_at_1_max
value: 29.761953273353463
- type: nauc_recall_at_1_std
value: -3.3283748548746397
- type: nauc_recall_at_20_diff1
value: 35.30332696049941
- type: nauc_recall_at_20_max
value: 38.8152320021852
- type: nauc_recall_at_20_std
value: 5.554654053834975
- type: nauc_recall_at_3_diff1
value: 43.74124312901872
- type: nauc_recall_at_3_max
value: 33.409247871088986
- type: nauc_recall_at_3_std
value: -2.3732020261032707
- type: nauc_recall_at_5_diff1
value: 41.29196881052914
- type: nauc_recall_at_5_max
value: 37.29462078993601
- type: nauc_recall_at_5_std
value: 2.2929109906730765
- type: ndcg_at_1
value: 46.595
- type: ndcg_at_10
value: 51.273
- type: ndcg_at_100
value: 57.379999999999995
- type: ndcg_at_1000
value: 59.602
- type: ndcg_at_20
value: 53.931
- type: ndcg_at_3
value: 46.678
- type: ndcg_at_5
value: 47.865
- type: precision_at_1
value: 46.595
- type: precision_at_10
value: 12.136
- type: precision_at_100
value: 1.73
- type: precision_at_1000
value: 0.20400000000000001
- type: precision_at_20
value: 7.016
- type: precision_at_3
value: 28.438000000000002
- type: precision_at_5
value: 20.401
- type: recall_at_1
value: 29.424
- type: recall_at_10
value: 59.666
- type: recall_at_100
value: 83.22
- type: recall_at_1000
value: 98.151
- type: recall_at_20
value: 68.195
- type: recall_at_3
value: 45.189
- type: recall_at_5
value: 51.32
task:
type: Retrieval
CATIE-AQ/distilcamembert-base-embedding
Description
This is a sentence-transformers model finetuned from cmarkea/distilcamembert-base (68.1M parameters). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Score on the MTEB leaderboard:
| Model | Average | Classification | Clustering | PairClassification | Reranking | Retrieval | STS | Summarization |
|---|---|---|---|---|---|---|---|---|
| CATIE-AQ/camembert-base-embedding (111M) | 60,057 | 66,117 | 45,41 | 79,675 | 71,303 | 45,769 | 82,049 | 30,074 |
| CATIE-AQ/distilcamembert-base-embedding (68M) | 58,297 | 63,904 | 44,549 | 79,102 | 67,961 | 42,222 | 80,204 | 30,138 |
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: cmarkea/distilcamembert-base
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: CamembertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("CATIE-AQ/distilcamembert-base-embedding")
# Run inference
sentences = [
"Tenet est sous surveillance depuis novembre, lorsque l'ancien directeur général Jeffrey Barbakow a déclaré que la société a utilisé des prix agressifs pour déclencher des paiements plus élevés pour les patients les plus malades de l'assurance maladie.",
"En novembre, Jeffrey Brabakow, le directeur général de l'époque, a déclaré que la société utilisait des prix agressifs pour obtenir des paiements plus élevés pour les patients les plus malades de l'assurance maladie.",
'La femme est en route pour un rendez-vous.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Citation
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}