Hugging Face's logo

aaa961
/
finetuned-bge-base-en-firefox

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
dense
Generated from Trainer
dataset_size:5424
loss:BatchSemiHardTripletLoss
Eval Results
text-embeddings-inference
Model card Files
xet
Community

SentenceTransformer based on BAAI/bge-base-en

This is a sentence-transformers model finetuned from BAAI/bge-base-en. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-base-en
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aaa961/finetuned-bge-base-en-firefox")
# Run inference
sentences = [
    "Loss of bookmarks state after OS hibernation User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0\n\nSteps to reproduce:\n\nKept Firefox 88.0 open for several days, using hibernate on Windows inbetween.\nCreated several bookmarks using Ctrl-D and Shift-Ctrl-D both for private as well as non-private sites.\nAlso created some bookmarks by dragging from address bar into subfolders of bookmark window.\nI was wondering, that on some windows star icon an address bar did not become blue.\nVerified, that i already had bookmarked this site by pressing Ctrl-D. The appropriate bookmarks subfolder was shown. Star still did not become true.\nAnything else seem ok.\nDid quit Firefox using File->Quit menu command, saving session with all open tabs.\nDid start Firefox with restoring previous session using menu command.\n\n\nActual results:\n\nSome of the bookmarks created above are lost. Can't remember most of them.\nHowever i remember a few sites, which i definitely had created a bookmark for.\nNo bookmark exists.\nAlso there is no entry in history for these sites. Missing entry in history might be due to using private window. (Can't remember if i was using a private window for opening this site.)\n\nI repeatedly had lost both bookmarks and history entries some month ago and then created a new profile. Did not import anything from previous profiles. \n\n\nExpected results:\n\nBookmarks should have been saved as expected.",
    'Messages with triggers cause trigger listeners to initialize multiple times [Currently](https://searchfox.org/mozilla-central/rev/07342ce09126c513540c1c343476e026cfa907bf/browser/components/newtab/lib/ASRouter.jsm#827) we call the `.init` function of every trigger (when defined).\nThis makes sense for `openURL` where we accumulate more URLs but for others it is a bug\n* moments trigger will add multiple setIntervals\n* whatsNew will evaluate multiple times which is wasteful).',
    'Urlbar docs: In index.rst, the Architecture Overview link in the Where to Start section is broken In index.rst, the Architecture Overview link in the Where to Start section links to an unrelated page in the devtools docs.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 1.0000],
#         [1.0000, 1.0000, 1.0000],
#         [1.0000, 1.0000, 1.0000]])

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.4904

Triplet

Metric Value
cosine_accuracy 0.5142

Training Details

Training Dataset

Unnamed Dataset

  • Size: 5,424 training samples
  • Columns: texts and label
  • Approximate statistics based on the first 1000 samples:
    texts label
    type string int
    details
    • min: 17 tokens
    • mean: 211.84 tokens
    • max: 511 tokens
    • 1: ~26.20%
    • 2: ~21.70%
    • 3: ~42.90%
    • 4: ~0.80%
    • 5: ~8.40%
  • Samples:
    texts label
    The "X" button of the opt-in modal is only read as "button" using a screen reader [Affected Versions]:
    - Firefox Beta 97.0b2 (Build ID: 20220111185943)
    - Firefox Nightly 98.a1 (Build ID: 20220111093827)

    [Affected Platforms]:
    - Windows 10 x64
    - Ubuntu 20.04 x64
    - macOS 10.15.7

    [Prerequisites]:
    - Have Firefox Beta 97.0b2 downloaded on your computer.
    - Have the "browser.search.region" set to "US".
    - Have one of the treatment user.js on your computer.
    - Make sure there is no other modal displayed when starting the browser (browser default window, onboarding for new users etc).
    - Have a screen reader application opened.

    [Steps to reproduce]:
    1. Open Firefox Beta 97.0b2.
    2. Navigate to the “about:support” page and paste the user.js file into the Profile folder.
    3. Restart the browser.
    4. Focus on the "X" button of the modal.
    5. Listen to what the screen reader application reads.

    **[Exp...    		 1
    Text and radio buttons are overlapping on PDF Tested with:
    Nightly 91.0a1 (2021-06-23)

    Tested on:
    Win 10

    Preconditions:
    In about:config, set pdfjs.enableXfa = true

    Steps:

    1. Launch Firefox
    2. Open the attached pdf.
    3. Go to "Taille de l'entreprise"

    Actual result:
    Radio buttons and text are overlapping.

    Expected result:
    Text and radio buttons should be properly displayed    		 1
    opening a new private window shows the old private window start page for a moment with browser.privatebrowsing.felt-privacy-v1, causing a bad perceived performance STR:

    1. Set browser.privatebrowsing.felt-privacy-v1 to true
    2. Open a private window

    Actual:

    The content of the old private window start page is visible during the window creation. This causes a very visible jump of the content and the window creation feels slow. Please see the following screencast:

    https://www.youtube.com/watch?v=Fax2VqefuuY

    You can change the speed to 0,25 on YouTube to see it even better.

    Expected:

    No felt performance issue.    		 3
  • Loss: BatchSemiHardTripletLoss

Evaluation Dataset

Unnamed Dataset

  • Size: 1,162 evaluation samples
  • Columns: texts and label
  • Approximate statistics based on the first 1000 samples:
    texts label
    type string int
    details
    • min: 16 tokens
    • mean: 209.86 tokens
    • max: 508 tokens
    • 1: ~26.30%
    • 2: ~21.00%
    • 3: ~42.50%
    • 4: ~0.70%
    • 5: ~9.50%
  • Samples:
    texts label
    (Proton) (a11y) Selected tabs almost not to see STR:

    1. select multiple tabs

    Expected:

    You have no difficulties to see your selected tabs.

    Actual:

    It's almost not possible (at least to me) to see the selected tabs due to a insufficient contrast between the tab bar color and the tab color.

    Marking as regression since it has a11y implications compared to pre Proton.    		 2
    The Not Now button from the Fakespot Onboarding sidebar is missing the Clicked State Found in
    * Nightly 118.0a1 (2023-08-18)

    Affected versions
    * Nightly 118.0a1 (2023-08-18)

    Affected platforms
    * ALL

    Preconditions:
    Set the browser.shopping.experience2023.enabled - TRUE
    Set the toolkit.shopping.useOHTTP - TRUE

    Steps to reproduce
    1. Reach about:preferences and turn off feature recommendations.
    2. Reach the Amazon https://www.amazon.com/dp/B09B6ZXD2V/ref=sbl_dpx_office-desks_B0B4CYW8FB_0 link
    3. Click and Hold the Not Now button from the Onboarding Shopping sidebar.

    Expected result
    * The Not now button from the Onboarding Shopping sidebar should change its state when Clicked.

    Actual result
    * The Not now button from the Onboarding Shopping sidebar is missing the Clicked state.

    Regression range
    Not Applicable    		 1
    Missing data from table Tested with:
    Nightly 91.0a1 (2021-06-22)

    Tested on:
    Win 10

    Preconditions:
    In about:config, set pdfjs.enableXfa = true

    Steps:

    1. Launch firefox
    2. Open the attached PDF

    Actual result:
    No data in table is displayed

    Expected result:
    A table with data should be displayed    		 1
  • Loss: BatchSemiHardTripletLoss

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 2
  • per_device_eval_batch_size: 2
  • gradient_accumulation_steps: 8
  • learning_rate: 2e-05
  • num_train_epochs: 5
  • warmup_ratio: 0.1
  • fp16: True
  • batch_sampler: group_by_label

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 2
  • per_device_eval_batch_size: 2
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 8
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 5
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: group_by_label
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss Validation Loss bge-base-en-train_cosine_accuracy
0.2950 100 5.4327 5.0655 0.5050
0.5900 200 5.2042 5.0281 0.4919
0.8850 300 5.1393 5.0186 0.5009
1.1829 400 5.1552 5.0137 0.5004
1.4779 500 5.0807 5.0111 0.4989
1.7729 600 5.0634 5.0092 0.4982
2.0708 700 5.1017 5.0079 0.4985
2.3658 800 5.0431 5.0067 0.4913
2.6608 900 5.0378 5.0058 0.4888
2.9558 1000 5.0332 5.0051 0.4904
3.2537 1100 5.08 5.0048 0.4897
3.5487 1200 5.0277 5.0043 0.4899
3.8437 1300 5.0257 5.0041 0.4888
4.1416 1400 5.075 5.0039 0.4924
4.4366 1500 5.0235 5.0038 0.4937
4.7316 1600 5.0224 5.0038 0.4904
-1 -1 - - 0.5142

Framework Versions

  • Python: 3.10.10
  • Sentence Transformers: 5.1.0
  • Transformers: 4.55.3
  • PyTorch: 2.7.1+cu128
  • Accelerate: 1.10.0
  • Datasets: 4.0.0
  • Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

BatchSemiHardTripletLoss 

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
Downloads last month
3
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for aaa961/finetuned-bge-base-en-firefox

Base model

BAAI/bge-base-en
Finetuned
(37)
this model

Evaluation results

View on Papers With Code