tomaarsen's picture
tomaarsen HF Staff
Add new SparseEncoder model
ce2f815 verified
metadata
language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sparse-encoder
  - sparse
  - splade
  - generated_from_trainer
  - dataset_size:99000
  - loss:SpladeLoss
  - loss:SparseMultipleNegativesRankingLoss
  - loss:FlopsLoss
base_model: google-bert/bert-base-uncased
widget:
  - text: >-
      The term emergent literacy signals a belief that, in a literate society,
      young children even one and two year olds, are in the process of becoming
      literate”. ... Gray (1956:21) notes: Functional literacy is used for the
      training of adults to 'meet independently the reading and writing demands
      placed on them'.
  - text: >-
      Rey is seemingly confirmed as being The Chosen One per a quote by a
      Lucasfilm production designer who worked on The Rise of Skywalker.
  - text: are union gun safes fireproof?
  - text: >-
      Fruit is an essential part of a healthy diet — and may aid weight loss.
      Most fruits are low in calories while high in nutrients and fiber, which
      can boost your fullness. Keep in mind that it's best to eat fruits whole
      rather than juiced. What's more, simply eating fruit is not the key to
      weight loss.
  - text: >-
      Treatment of suspected bacterial infection is with antibiotics, such as
      amoxicillin/clavulanate or doxycycline, given for 5 to 7 days for acute
      sinusitis and for up to 6 weeks for chronic sinusitis.
datasets:
  - sentence-transformers/gooaq
pipeline_tag: feature-extraction
library_name: sentence-transformers
metrics:
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
  - query_active_dims
  - query_sparsity_ratio
  - corpus_active_dims
  - corpus_sparsity_ratio
co2_eq_emissions:
  emissions: 22.555839031832004
  energy_consumed: 0.058028615833805856
  source: codecarbon
  training_type: fine-tuning
  on_cloud: false
  cpu_model: 13th Gen Intel(R) Core(TM) i7-13700K
  ram_total_size: 31.777088165283203
  hours_used: 0.232
  hardware_used: 1 x NVIDIA GeForce RTX 3090
model-index:
  - name: bert-base-uncased adapter finetuned on GooAQ pairs
    results:
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoMSMARCO
          type: NanoMSMARCO
        metrics:
          - type: dot_accuracy@1
            value: 0.16
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.4
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.52
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.6
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.16
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.13333333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.10400000000000001
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.06000000000000001
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.16
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.4
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.52
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.6
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.38421491435302385
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.3143571428571429
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.33474113801530114
            name: Dot Map@100
          - type: query_active_dims
            value: 134.27999877929688
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9956005504626402
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 569.0715942382812
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9813553635332455
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.18
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.42
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.5
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.66
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.18
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.13999999999999999
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.1
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.066
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.18
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.42
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.5
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.66
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.40466314485393395
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.32460317460317456
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.3399207622442916
            name: Dot Map@100
          - type: query_active_dims
            value: 119.91999816894531
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9960710307919224
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 558.7083129882812
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9816948983360106
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoNFCorpus
          type: NanoNFCorpus
        metrics:
          - type: dot_accuracy@1
            value: 0.28
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.38
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.48
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.56
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.28
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.21333333333333332
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.20799999999999996
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.18999999999999997
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.009643842973249836
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.021145927666063272
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.03330669347394032
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.06056852476443145
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.20300898862073832
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.35271428571428565
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.06784215422201557
            name: Dot Map@100
          - type: query_active_dims
            value: 225.1999969482422
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9926217155838988
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 758.9915161132812
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9751329691333045
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.28
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.4
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.5
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.52
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.28
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.21333333333333332
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.204
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.192
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.009643842973249836
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.020713027233162838
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.033373623471745904
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.057702524164229184
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.20494533130049183
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.3591904761904762
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.07033841029751142
            name: Dot Map@100
          - type: query_active_dims
            value: 204.0800018310547
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9933136753217006
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 768.1585083007812
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.974832628651439
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoNQ
          type: NanoNQ
        metrics:
          - type: dot_accuracy@1
            value: 0.28
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.52
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.62
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.68
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.28
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.1733333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.12400000000000003
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.07
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.27
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.48
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.58
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.64
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.456183710090412
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.41005555555555545
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.40230708079647753
            name: Dot Map@100
          - type: query_active_dims
            value: 115.80000305175781
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9962060152332167
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 666.47705078125
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9781640439426887
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.26
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.46
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.62
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.72
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.26
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.15333333333333332
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.12400000000000003
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.07400000000000001
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.25
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.43
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.58
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.68
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.4523767145039781
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.39466666666666655
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.3832114962839765
            name: Dot Map@100
          - type: query_active_dims
            value: 104.4800033569336
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9965768952441867
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 660.2720947265625
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9783673384861227
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-nano-beir
          name: Sparse Nano BEIR
        dataset:
          name: NanoBEIR mean
          type: NanoBEIR_mean
        metrics:
          - type: dot_accuracy@1
            value: 0.24000000000000002
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.43333333333333335
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.54
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.6133333333333334
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.24000000000000002
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.17333333333333334
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.14533333333333331
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.10666666666666667
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.14654794765774995
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.30038197588868776
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.37776889782464673
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.4335228415881438
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.347802537688058
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.359042328042328
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.26829679101126475
            name: Dot Map@100
          - type: query_active_dims
            value: 158.42666625976562
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9948094270932519
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 649.7461397828076
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9787122030082299
            name: Corpus Sparsity Ratio
          - type: dot_accuracy@1
            value: 0.30251177394034545
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.46596546310832027
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.539905808477237
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.6323076923076923
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.30251177394034545
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.21529042386185243
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.1783736263736264
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.13637676609105182
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.14225413356767205
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.2584890881168072
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.3166858200553643
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.402504217163254
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.352844324981004
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.40464005382372714
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.28418762466289216
            name: Dot Map@100
          - type: query_active_dims
            value: 274.27427396495096
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9910138826431769
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 613.3641760134007
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9799041944822292
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoClimateFEVER
          type: NanoClimateFEVER
        metrics:
          - type: dot_accuracy@1
            value: 0.18
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.32
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.36
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.5
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.18
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.12
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.092
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.07
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.09666666666666666
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.15899999999999997
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.19066666666666665
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.26566666666666666
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.21060314835282243
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.26110317460317456
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.16539103175017164
            name: Dot Map@100
          - type: query_active_dims
            value: 464.260009765625
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9847893319649557
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 703.13818359375
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9769629059827747
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoDBPedia
          type: NanoDBPedia
        metrics:
          - type: dot_accuracy@1
            value: 0.54
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.7
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.84
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.88
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.54
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.45333333333333337
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.45199999999999996
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.414
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.04495371619535625
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.09854649158620489
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.14811311296402138
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.2518191059637309
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.47484851291171687
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.657079365079365
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.333901271745345
            name: Dot Map@100
          - type: query_active_dims
            value: 190.66000366210938
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9937533581134228
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 732.0421752929688
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9760159171976618
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoFEVER
          type: NanoFEVER
        metrics:
          - type: dot_accuracy@1
            value: 0.34
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.56
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.58
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.7
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.34
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.19333333333333336
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.12000000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.07200000000000001
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.32
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.55
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.57
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.6766666666666667
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.5066164614256192
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.460547619047619
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.4600294644045045
            name: Dot Map@100
          - type: query_active_dims
            value: 396.8800048828125
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.986996920094266
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 838.1881713867188
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9725382291007563
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoFiQA2018
          type: NanoFiQA2018
        metrics:
          - type: dot_accuracy@1
            value: 0.16
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.28
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.36
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.46
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.16
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.11999999999999998
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.11200000000000002
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.07600000000000001
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.06655555555555555
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.16722222222222222
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.2236031746031746
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.32060317460317456
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.22703772428242097
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.24921428571428572
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.17973302435608687
            name: Dot Map@100
          - type: query_active_dims
            value: 137.83999633789062
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9954839133628893
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 472.53118896484375
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9845183412304289
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoHotpotQA
          type: NanoHotpotQA
        metrics:
          - type: dot_accuracy@1
            value: 0.64
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.78
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.84
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.96
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.64
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.32
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.22799999999999998
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.136
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.32
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.48
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.57
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.68
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.5959463797348153
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.7347142857142858
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.5051629575345892
            name: Dot Map@100
          - type: query_active_dims
            value: 147.22000122070312
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9951765938922514
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 793.5137329101562
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9740019090193908
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoQuoraRetrieval
          type: NanoQuoraRetrieval
        metrics:
          - type: dot_accuracy@1
            value: 0.2
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.36
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.42
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.48
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.2
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.13333333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.092
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.052000000000000005
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.17666666666666667
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.32733333333333337
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.38733333333333336
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.434
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.31674229187216874
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.29157142857142854
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.29472815560103693
            name: Dot Map@100
          - type: query_active_dims
            value: 77
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9974772295393487
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 72.00594329833984
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9976408510812419
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoSCIDOCS
          type: NanoSCIDOCS
        metrics:
          - type: dot_accuracy@1
            value: 0.18
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.38
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.42
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.6
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.18
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.16666666666666669
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.11599999999999999
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.09200000000000001
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.036000000000000004
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.10100000000000002
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.117
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.18866666666666668
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.1725314970247529
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.2993571428571428
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.1316070110116111
            name: Dot Map@100
          - type: query_active_dims
            value: 390.1000061035156
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.98721905490782
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 708.0321655273438
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9768025632158004
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoArguAna
          type: NanoArguAna
        metrics:
          - type: dot_accuracy@1
            value: 0.06
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.14
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.16
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.18
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.06
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.04666666666666667
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.032
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.018000000000000002
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.06
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.14
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.16
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.18
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.11875554281390689
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.09933333333333332
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.10937808986204274
            name: Dot Map@100
          - type: query_active_dims
            value: 699.3400268554688
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9770873459519209
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 637.1741333007812
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9791241028339958
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoSciFact
          type: NanoSciFact
        metrics:
          - type: dot_accuracy@1
            value: 0.28
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.38
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.48
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 0.56
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.28
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.13333333333333333
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.10400000000000001
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.066
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.245
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.345
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.455
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.55
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.39370005539130326
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.3605238095238095
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.34373433911333495
            name: Dot Map@100
          - type: query_active_dims
            value: 591.9000244140625
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.9806074299058364
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 700.0397338867188
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9770644212736151
            name: Corpus Sparsity Ratio
      - task:
          type: sparse-information-retrieval
          name: Sparse Information Retrieval
        dataset:
          name: NanoTouche2020
          type: NanoTouche2020
        metrics:
          - type: dot_accuracy@1
            value: 0.6326530612244898
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.8775510204081632
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9387755102040817
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 1
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.6326530612244898
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.6054421768707483
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.5428571428571428
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.4448979591836735
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.04381728832224134
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.12154307114357087
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.18182574968079382
            name: Dot Recall@5
          - type: dot_recall@10
            value: 0.28743001839116805
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.5082094202851214
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.768415937803693
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.37730310641309606
            name: Dot Map@100
          - type: query_active_dims
            value: 37.14285659790039
            name: Query Active Dims
          - type: query_sparsity_ratio
            value: 0.998783079201956
            name: Query Sparsity Ratio
          - type: corpus_active_dims
            value: 505.10235595703125
            name: Corpus Active Dims
          - type: corpus_sparsity_ratio
            value: 0.9834512038543664
            name: Corpus Sparsity Ratio

bert-base-uncased adapter finetuned on GooAQ pairs

This is a SPLADE Sparse Encoder model finetuned from google-bert/bert-base-uncased on the gooaq dataset using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

  • Model Type: SPLADE Sparse Encoder
  • Base model: google-bert/bert-base-uncased
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 30522 dimensions
  • Similarity Function: Dot Product
  • Training Dataset:
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("tomaarsen/splade-bert-base-uncased-gooaq-peft")
# Run inference
queries = [
    "how many days for doxycycline to work on sinus infection?",
]
documents = [
    'Treatment of suspected bacterial infection is with antibiotics, such as amoxicillin/clavulanate or doxycycline, given for 5 to 7 days for acute sinusitis and for up to 6 weeks for chronic sinusitis.',
    'Most engagements typically have a cocktail dress code, calling for dresses at, or slightly above, knee-length and high heels. If your party states a different dress code, however, such as semi-formal or dressy-casual, you may need to dress up or down accordingly.',
    'The average service life of a gas furnace is about 15 years, but the actual life span of an individual unit can vary greatly. There are a number of contributing factors that determine the age a furnace reaches: The quality of the equipment.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[90.9602, 31.2714, 39.4626]])

Evaluation

Metrics

Sparse Information Retrieval

  • Datasets: NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
  • Evaluated with SparseInformationRetrievalEvaluator
Metric NanoMSMARCO NanoNFCorpus NanoNQ NanoClimateFEVER NanoDBPedia NanoFEVER NanoFiQA2018 NanoHotpotQA NanoQuoraRetrieval NanoSCIDOCS NanoArguAna NanoSciFact NanoTouche2020
dot_accuracy@1 0.18 0.28 0.26 0.18 0.54 0.34 0.16 0.64 0.2 0.18 0.06 0.28 0.6327
dot_accuracy@3 0.42 0.4 0.46 0.32 0.7 0.56 0.28 0.78 0.36 0.38 0.14 0.38 0.8776
dot_accuracy@5 0.5 0.5 0.62 0.36 0.84 0.58 0.36 0.84 0.42 0.42 0.16 0.48 0.9388
dot_accuracy@10 0.66 0.52 0.72 0.5 0.88 0.7 0.46 0.96 0.48 0.6 0.18 0.56 1.0
dot_precision@1 0.18 0.28 0.26 0.18 0.54 0.34 0.16 0.64 0.2 0.18 0.06 0.28 0.6327
dot_precision@3 0.14 0.2133 0.1533 0.12 0.4533 0.1933 0.12 0.32 0.1333 0.1667 0.0467 0.1333 0.6054
dot_precision@5 0.1 0.204 0.124 0.092 0.452 0.12 0.112 0.228 0.092 0.116 0.032 0.104 0.5429
dot_precision@10 0.066 0.192 0.074 0.07 0.414 0.072 0.076 0.136 0.052 0.092 0.018 0.066 0.4449
dot_recall@1 0.18 0.0096 0.25 0.0967 0.045 0.32 0.0666 0.32 0.1767 0.036 0.06 0.245 0.0438
dot_recall@3 0.42 0.0207 0.43 0.159 0.0985 0.55 0.1672 0.48 0.3273 0.101 0.14 0.345 0.1215
dot_recall@5 0.5 0.0334 0.58 0.1907 0.1481 0.57 0.2236 0.57 0.3873 0.117 0.16 0.455 0.1818
dot_recall@10 0.66 0.0577 0.68 0.2657 0.2518 0.6767 0.3206 0.68 0.434 0.1887 0.18 0.55 0.2874
dot_ndcg@10 0.4047 0.2049 0.4524 0.2106 0.4748 0.5066 0.227 0.5959 0.3167 0.1725 0.1188 0.3937 0.5082
dot_mrr@10 0.3246 0.3592 0.3947 0.2611 0.6571 0.4605 0.2492 0.7347 0.2916 0.2994 0.0993 0.3605 0.7684
dot_map@100 0.3399 0.0703 0.3832 0.1654 0.3339 0.46 0.1797 0.5052 0.2947 0.1316 0.1094 0.3437 0.3773
query_active_dims 119.92 204.08 104.48 464.26 190.66 396.88 137.84 147.22 77.0 390.1 699.34 591.9 37.1429
query_sparsity_ratio 0.9961 0.9933 0.9966 0.9848 0.9938 0.987 0.9955 0.9952 0.9975 0.9872 0.9771 0.9806 0.9988
corpus_active_dims 558.7083 768.1585 660.2721 703.1382 732.0422 838.1882 472.5312 793.5137 72.0059 708.0322 637.1741 700.0397 505.1024
corpus_sparsity_ratio 0.9817 0.9748 0.9784 0.977 0.976 0.9725 0.9845 0.974 0.9976 0.9768 0.9791 0.9771 0.9835

Sparse Nano BEIR

  • Dataset: NanoBEIR_mean
  • Evaluated with SparseNanoBEIREvaluator with these parameters:
    {
        "dataset_names": [
            "msmarco",
            "nfcorpus",
            "nq"
        ]
    }
    
Metric Value
dot_accuracy@1 0.24
dot_accuracy@3 0.4333
dot_accuracy@5 0.54
dot_accuracy@10 0.6133
dot_precision@1 0.24
dot_precision@3 0.1733
dot_precision@5 0.1453
dot_precision@10 0.1067
dot_recall@1 0.1465
dot_recall@3 0.3004
dot_recall@5 0.3778
dot_recall@10 0.4335
dot_ndcg@10 0.3478
dot_mrr@10 0.359
dot_map@100 0.2683
query_active_dims 158.4267
query_sparsity_ratio 0.9948
corpus_active_dims 649.7461
corpus_sparsity_ratio 0.9787

Sparse Nano BEIR

  • Dataset: NanoBEIR_mean
  • Evaluated with SparseNanoBEIREvaluator with these parameters:
    {
        "dataset_names": [
            "climatefever",
            "dbpedia",
            "fever",
            "fiqa2018",
            "hotpotqa",
            "msmarco",
            "nfcorpus",
            "nq",
            "quoraretrieval",
            "scidocs",
            "arguana",
            "scifact",
            "touche2020"
        ]
    }
    
Metric Value
dot_accuracy@1 0.3025
dot_accuracy@3 0.466
dot_accuracy@5 0.5399
dot_accuracy@10 0.6323
dot_precision@1 0.3025
dot_precision@3 0.2153
dot_precision@5 0.1784
dot_precision@10 0.1364
dot_recall@1 0.1423
dot_recall@3 0.2585
dot_recall@5 0.3167
dot_recall@10 0.4025
dot_ndcg@10 0.3528
dot_mrr@10 0.4046
dot_map@100 0.2842
query_active_dims 274.2743
query_sparsity_ratio 0.991
corpus_active_dims 613.3642
corpus_sparsity_ratio 0.9799

Training Details

Training Dataset

gooaq

  • Dataset: gooaq at b089f72
  • Size: 99,000 training samples
  • Columns: question and answer
  • Approximate statistics based on the first 1000 samples:
    question answer
    type string string
    details
    • min: 8 tokens
    • mean: 11.79 tokens
    • max: 24 tokens
    • min: 14 tokens
    • mean: 60.02 tokens
    • max: 153 tokens
  • Samples:
    question answer
    what are the 5 characteristics of a star? Key Concept: Characteristics used to classify stars include color, temperature, size, composition, and brightness.
    are copic markers alcohol ink? Copic Ink is alcohol-based and flammable. Keep away from direct sunlight and extreme temperatures.
    what is the difference between appellate term and appellate division? Appellate terms An appellate term is an intermediate appellate court that hears appeals from the inferior courts within their designated counties or judicial districts, and are intended to ease the workload on the Appellate Division and provide a less expensive forum closer to the people.
  • Loss: SpladeLoss with these parameters:
    {
        "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
        "document_regularizer_weight": 3e-05,
        "query_regularizer_weight": 5e-05
    }
    

Evaluation Dataset

gooaq

  • Dataset: gooaq at b089f72
  • Size: 1,000 evaluation samples
  • Columns: question and answer
  • Approximate statistics based on the first 1000 samples:
    question answer
    type string string
    details
    • min: 8 tokens
    • mean: 11.93 tokens
    • max: 25 tokens
    • min: 14 tokens
    • mean: 60.84 tokens
    • max: 127 tokens
  • Samples:
    question answer
    should you take ibuprofen with high blood pressure? In general, people with high blood pressure should use acetaminophen or possibly aspirin for over-the-counter pain relief. Unless your health care provider has said it's OK, you should not use ibuprofen, ketoprofen, or naproxen sodium. If aspirin or acetaminophen doesn't help with your pain, call your doctor.
    how old do you have to be to work in sc? The general minimum age of employment for South Carolina youth is 14, although the state allows younger children who are performers to work in show business. If their families are agricultural workers, children younger than age 14 may also participate in farm labor.
    how to write a topic proposal for a research paper? ['Write down the main topic of your paper. ... ', 'Write two or three short sentences under the main topic that explain why you chose that topic. ... ', 'Write a thesis sentence that states the angle and purpose of your research paper. ... ', 'List the items you will cover in the body of the paper that support your thesis statement.']
  • Loss: SpladeLoss with these parameters:
    {
        "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
        "document_regularizer_weight": 3e-05,
        "query_regularizer_weight": 5e-05
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • learning_rate: 2e-05
  • num_train_epochs: 1
  • bf16: True
  • load_best_model_at_end: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss Validation Loss NanoMSMARCO_dot_ndcg@10 NanoNFCorpus_dot_ndcg@10 NanoNQ_dot_ndcg@10 NanoBEIR_mean_dot_ndcg@10 NanoClimateFEVER_dot_ndcg@10 NanoDBPedia_dot_ndcg@10 NanoFEVER_dot_ndcg@10 NanoFiQA2018_dot_ndcg@10 NanoHotpotQA_dot_ndcg@10 NanoQuoraRetrieval_dot_ndcg@10 NanoSCIDOCS_dot_ndcg@10 NanoArguAna_dot_ndcg@10 NanoSciFact_dot_ndcg@10 NanoTouche2020_dot_ndcg@10
0.0323 100 91.8917 - - - - - - - - - - - - - - -
0.0646 200 8.1616 - - - - - - - - - - - - - - -
0.0970 300 1.6108 - - - - - - - - - - - - - - -
0.1293 400 0.7834 - - - - - - - - - - - - - - -
0.1616 500 0.5234 - - - - - - - - - - - - - - -
0.1939 600 0.396 - - - - - - - - - - - - - - -
0.1972 610 - 0.3011 0.3418 0.1789 0.2676 0.2628 - - - - - - - - - -
0.2262 700 0.4065 - - - - - - - - - - - - - - -
0.2586 800 0.3367 - - - - - - - - - - - - - - -
0.2909 900 0.3305 - - - - - - - - - - - - - - -
0.3232 1000 0.3454 - - - - - - - - - - - - - - -
0.3555 1100 0.284 - - - - - - - - - - - - - - -
0.3878 1200 0.2639 - - - - - - - - - - - - - - -
0.3943 1220 - 0.2610 0.4244 0.2060 0.3846 0.3383 - - - - - - - - - -
0.4202 1300 0.2787 - - - - - - - - - - - - - - -
0.4525 1400 0.2343 - - - - - - - - - - - - - - -
0.4848 1500 0.2833 - - - - - - - - - - - - - - -
0.5171 1600 0.2706 - - - - - - - - - - - - - - -
0.5495 1700 0.2277 - - - - - - - - - - - - - - -
0.5818 1800 0.2515 - - - - - - - - - - - - - - -
0.5915 1830 - 0.2282 0.3836 0.1977 0.3801 0.3204 - - - - - - - - - -
0.6141 1900 0.1985 - - - - - - - - - - - - - - -
0.6464 2000 0.2224 - - - - - - - - - - - - - - -
0.6787 2100 0.1904 - - - - - - - - - - - - - - -
0.7111 2200 0.2564 - - - - - - - - - - - - - - -
0.7434 2300 0.2333 - - - - - - - - - - - - - - -
0.7757 2400 0.208 - - - - - - - - - - - - - - -
0.7886 2440 - 0.2087 0.4047 0.2049 0.4524 0.354 - - - - - - - - - -
0.8080 2500 0.1874 - - - - - - - - - - - - - - -
0.8403 2600 0.2296 - - - - - - - - - - - - - - -
0.8727 2700 0.1926 - - - - - - - - - - - - - - -
0.9050 2800 0.2023 - - - - - - - - - - - - - - -
0.9373 2900 0.2098 - - - - - - - - - - - - - - -
0.9696 3000 0.1607 - - - - - - - - - - - - - - -
0.9858 3050 - 0.2088 0.3842 0.2030 0.4562 0.3478 - - - - - - - - - -
-1 -1 - - 0.4047 0.2049 0.4524 0.3528 0.2106 0.4748 0.5066 0.2270 0.5959 0.3167 0.1725 0.1188 0.3937 0.5082
  • The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

  • Energy Consumed: 0.058 kWh
  • Carbon Emitted: 0.023 kg of CO2
  • Hours Used: 0.232 hours

Training Hardware

  • On Cloud: No
  • GPU Model: 1 x NVIDIA GeForce RTX 3090
  • CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
  • RAM Size: 31.78 GB

Framework Versions

  • Python: 3.11.6
  • Sentence Transformers: 4.2.0.dev0
  • Transformers: 4.52.4
  • PyTorch: 2.7.1+cu126
  • Accelerate: 1.5.1
  • Datasets: 2.21.0
  • Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}