sch-allie/bert-persona-model

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'I am a vegan, I like to go for bike rides, I like the guitar, I like to skateboard, My major is in business administration, I work at a daycare, My dad is a dentist and my mom is a teacher, I am a college student, I have been in a relationship for 2 years',
    'I am a writer, I dream of becoming a famous actress, I don t like the beach, Zebras are my favorite animals, My newly bought laptop has a bum battery, I write short stories in my document tab with the use of a prompting app, My college courses are on philosophy and history, My glasses are held together by tiger printed duct tape',
    'I am an orphan, I like dogs, I now live in new mexico, I grew up in nevada, I went to miami university, I make a million dollars a year, I play for the baltimore orioles, I m married and have three kids, I am a baseball player',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

• Evaluated with BinaryClassificationEvaluator

Metric	Value
cosine_accuracy	0.9674
cosine_accuracy_threshold	0.7207
cosine_f1	0.8128
cosine_f1_threshold	0.699
cosine_precision	0.7666
cosine_recall	0.8648
cosine_ap	0.8465
cosine_mcc	0.7962

Training Details

Training Dataset

Unnamed Dataset

• Size: 53,796 training samples
• Columns: persona1, persona2, and label
• Approximate statistics based on the first 1000 samples:

  	persona1	persona2	label
  type	string	string	int
  details	min: 38 tokens mean: 63.1 tokens max: 121 tokens	min: 40 tokens mean: 63.57 tokens max: 103 tokens	0: ~93.30% 1: ~6.70%

• Samples:

  persona1	persona2	label
  I love dancing, I love country music, I am a vegan, I love animals, I am a poet, My friends and i enjoy golfing on sunny days, I am the head of the gun club, I have a beta fish, I donate old clothes to the homeless	I like antiques, I like jazz, I am a romantic, I collect dolls, I like victorian things, I i love logical and rational thinking, I am very good at math and science, I am considered a nerd by many, I started working at google last week on self driving car research, I love computers	0
  I like to paint, We have two sons, I enjoy visiting museums, I like to attend wine tours, My husband is 20 years older than me, I love to read, I learned how to reads when i was three, I can read in english and french, I read three books a week, I dropped out of high school	I m a vegan, I have a tattoo of an angel on my hip, My eyes are brown, I study philosophy at umass, I love going to concerts, I am an athlete, I race cars for a living, I have 4 daughters, I like to go fishing, I like to play board games	0
  I am a nurse, I was an army brat, I married my high school sweetheart, I surf often, I am a great baker, I am a social butterfly, I like to swim, I am in college, I exercise everyday, I eat large meals	I surf often, I married my high school sweetheart, I am a nurse, I am a great baker, I was an army brat, I have two kids , ages 2 and 6, My husband owns a small auto repair shop, I work part time at aldi s, My favorite movie is titanic, I am from sterling heights , michigan	1

• Loss: ContrastiveLoss with these parameters:

  {
      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
      "margin": 0.5,
      "size_average": true
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 13,450 evaluation samples
• Columns: persona1, persona2, and label
• Approximate statistics based on the first 1000 samples:

  	persona1	persona2	label
  type	string	string	int
  details	min: 36 tokens mean: 62.92 tokens max: 112 tokens	min: 36 tokens mean: 62.89 tokens max: 121 tokens	0: ~91.70% 1: ~8.30%

• Samples:

  persona1	persona2	label
  I have two dogs, I lease my car, I work in accounting, I like potatoes, I am male, I speak fluent italian, I have one blue eye and one hazel eye, My brother is an archaeologist, I am a cat owner, My mom is an osteopath	I listen to rap, I drive a black car, My favorite food is steak, I like meat, I am working in finance, I like to role play, My favorite food is pizza, I love cars, I do not like animals	0
  I have two dogs, I like to cook, I m scared of clowns, I have two roomates, My hair is a reddish brown, Although i ski down high hills , i have a fear of heights, I am a competitive ski racer, My family is very supportive of my skiing	I love cats, I decorate cakes for a living, I play a lot of video games, I am a lesbian, I like to read books that are in a series, I like to take drives in the country, I love to go out to eat with my family, I like to chat with my friends, I like to go to the movies	0
  I love the rain, I prefer winter, I drive a van, My favorite food is pizza, My dream job is a to become a baseball announcer, I watch south park at least once a day, My favorite band is avenged sevenfold, I currently hold three jobs, I recently proposed to my girlfriend of three years	I am a paramedic, I have three sisters, I am studying to become a nurse, My favorite band is the beatles, I live in a studio apartment, I am from mexico, I used to be a chef , but i am a teacher now, I like to go on walks, I like to bake	0

• Loss: ContrastiveLoss with these parameters:

  {
      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
      "margin": 0.5,
      "size_average": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	cosine_ap
1.0	211	-	0.0051	0.8646
2.0	422	-	0.0045	0.8677
2.3697	500	0.0058	-	-
3.0	633	-	0.0052	0.8577
4.0	844	-	0.0047	0.8506
4.7393	1000	0.0045	-	-
5.0	1055	-	0.0047	0.8465

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 3.4.1
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveLoss

@inproceedings{hadsell2006dimensionality,
    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)},
    title={Dimensionality Reduction by Learning an Invariant Mapping},
    year={2006},
    volume={2},
    number={},
    pages={1735-1742},
    doi={10.1109/CVPR.2006.100}
}