Upload folder using huggingface_hub

.gitattributes

@@ -40,3 +40,9 @@ image-16.png filter=lfs diff=lfs merge=lfs -text
 image-18.png filter=lfs diff=lfs merge=lfs -text
 image-9.png filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/image-1.png filter=lfs diff=lfs merge=lfs -text
+assets/image-10.png filter=lfs diff=lfs merge=lfs -text
+assets/image-11.png filter=lfs diff=lfs merge=lfs -text
+assets/image-16.png filter=lfs diff=lfs merge=lfs -text
+assets/image-18.png filter=lfs diff=lfs merge=lfs -text
+assets/image-9.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,214 @@
+---
+license: apache-2.0
+tags:
+  - sentence-transformers
+  - sentence-similarity
+  - mteb
+  - retriever
+  - text-embeddings-inference
+---
+# QZhou-Embedding
+<div align="center">
+<img src="assets/image-1.png" width="800" height="300"></img>
+</div>
+
+## Introduction
+We have released <a href="https://huggingface.co/Kingsoft-LLM/QZhou-Embedding">QZhou-Embedding</a> (called "Qingzhou Embedding"), a large-scale text embedding model designed for general use，excelling at various text embedding tasks (retrieval, re-ranking, sentence similarity, and classification). Leveraging the general language capabilities of its underlying model, and pre-trained on massive amounts of text, QZhou-Embedding achieves even more powerful text embedding representations. QZhou-Embedding is continuously trained using millions of high-quality open-source embedding datasets and over 5 million high-quality synthetic data (using two synthetic techniques: rewriting and expansion). Initial retrieval training provides the model with a foundation for query-doc semantic matching capabilities. Later, multi-dimensional training such as STS and clustering, helps the model achieve continuous breakthroughs in various tasks. QZhou-Embedding is a 7B model and can embed long text vectors up to 8k in size. It achieved the highest average score on the mteb/cmteb evaluation benchmarks. In terms of various task scores, its clustering, sentence pair classification, rearrangement, and STS task achieved the highest average scores.
+## Basic Features
+
+- Powerful text embedding capabilities；
+- Long context: up to 8k context length；
+- 7B parameter size
+
+
+## Technical Introduction
+### Unified Task Modeling Framework
+We unify the text embedding objectives into three major modeling optimization issues and propose a unified training data structured solution and corresponding training mechanism. This approach can integrate most open source data as retrieval training sets. The structured data can be as follows:
+- Retrieval
+  - title-body
+  - title-abstract
+  - Question Answering Dataset
+  - Reading comprehension
+  - ...
+
+- STS
+  - text pair + label in {true, false}、{yes, no}
+  - text pair + score（such as 0.2, 3.1. 4.8, etc.）
+  - NLI dataset：text pair + label in {'entailment', 'neutral', 'contradiction'}
+
+- CLS
+  - text+CLS label
+
+<div align="center"><img src="assets/image-18.png" width="1000" height="600"></img></div>
+<div align="center"><img src="assets/image-16.png" width="1000" height="550"></img></div>
+
+### Training Objectives
+
+- Retrieval: Apply InfoNCE contrastive loss function, and follow the gte/qwen3-embedding to add the query-query negative as part of the denominator.<br>
+$$
+L_{ret}=-\frac{1}{n}\sum_{i} log{\frac{e^{sim(q_i,d_i^+)/\tau}}{e^{sim(q_i,d_i^+)/\tau}+\sum_{j}e^{sim(q_i,d_j^-)/\tau}+\sum_{j≠i}e^{sim(q_i,q_j)/\tau}}}
+$$
+
+- STS：Apply Cosent loss：
+$$
+L_{cosent}=log \bigg(1+\sum_{sim(i,j)>sim(k,l)}exp(\frac{sim(x_k, x_l)-sim(x_i,x_j)}{\tau})\bigg)
+$$
+
+- CLS: Apply the same InfoNCE loss as retrieval, but for In-Batch Negative, due to the high probability of same-class conflicts, a mask mechanism is used to cover up similar samples in negative examples shared by different samples.
+$$
+L_{ret}=-\frac{1}{n}\sum_{i} log{\frac{e^{sim(t_i,t_i^+)/\tau}}{e^{sim(t_i,t_i^+)/\tau}+\sum_{n}MASK(t_i,t_{i,n}^-)·e^{sim(t_i,t_{i,n}^-)/\tau}+\sum_{j≠i}MASK(t_i,t_j)·e^{sim(t_i,t_j)/\tau}+\sum_{j≠i}\sum_{n}MASK(t_i,t_{j,n}^-)e^{sim(t_i,t_{j,n}^-)/\tau}}}
+$$
+$$
+where\:\:C_{t_i}=C_{t_i^+}
+$$
+$$
+MASK(t_i, t_j)=
+\begin{cases}
+0 & \quad \text{if } C_{t_i}=C_{t_j}, \\
+1 & \quad \text{otherwise}
+\end{cases}
+$$
+Where $C_{t_i}$ represents the class label of sample $t_i$ , and $n$ is the number of negative samples for a single data point.
+### Feature Enhancement Data Synthesis Technology
+In the context of powerful languages and writing capabilities in LLMs, we've fully leveraged the LLMs API to propose a data synthesis technology. To address issues like limited data and narrow topics/features in training sets, we've proposed rewriting and expanding synthesis techniques. Furthermore, to increase the difficulty of negative examples during training, we've designed a hard negative example synthesis technology based on big models, combined with existing strong retriever-based hard negative examples sampling. Several of these technologies are described below:
+<div align="center"><img src="assets/image-9.png" width="930" height="290"></img></div>
+<div align="center"><img src="assets/image-10.png" width="880" height="220"></img></div>
+<div align="center"><img src="assets/image-11.png" width="880" height="210"></img></div>
+
+For more details, including reproduction of evaluation results, Instruction content and adding method, please refer to our <a href="https://github.com/Kingsoft-LLM/QZhou-Embedding">GitHub</a> repo, thanks! 
+
+## Evaluation Results
+### mteb details
+<div align="center"><img src="assets/image-7.png" width="1100" height="260"></img></div>
+
+### cmteb details
+<div align="center"><img src="assets/image-8.png" width="1000" height="260"></img></div>
+
+## Usage
+### Completely reproduce the benchmark results
+We provide detailed parameters and environment configurations so that you can run results that are completely consistent with the mteb leaderboard on your own machine, including configurations such as environment dependencies and model arguments.
+#### Requirements
+- Python: 3.10.12
+- Sentence Transformers: 3.4.1
+- Transformers: 4.51.1
+- PyTorch: 2.7.1
+- Accelerate: 1.3.0
+- Datasets: 3.2.0
+- Tokenizers: 0.21.2
+#### Transformers model load arguments
+torch_dtype=torch.bfloat16<br>
+attn_implementation='sdpa'<br>
+**NOTE：** The ranking results use the sdpa mode. Other modes ('eager', 'flash_attention_2') may have deviations in results, but still keep the overall performance consistent.
+#### Instruction Adding Rules
+Details can be found on our <a href="https://github.com/Kingsoft-LLM/QZhou-Embedding">GitHub</a>.
+#### Evaluation code usage
+Find our benchmark evaluation code on <a href="https://github.com/Kingsoft-LLM/QZhou-Embedding">GitHub</a>. The mteb benchmark script is **run_mteb_all_v2.py**, and the cmteb benchmark script is **run_cmteb_all.py**. Run the following command:
+```
+POOLING_MODE=mean
+normalize=true
+use_instruction=true
+export TOKENIZERS_PARALLELISM=true
+
+model_name_or_path=<model dir>
+
+python3 ./run_cmteb_all.py \
+    --model_name_or_path ${model_name_or_path}  \
+    --pooling_mode ${POOLING_MODE} \
+    --normalize ${normalize} \
+    --use_instruction ${use_instruction} \
+    --output_dir <output dir>
+
+python3 ./run_mteb_all_v2.py \
+    --model_name_or_path ${model_name_or_path}  \
+    --pooling_mode ${POOLING_MODE} \
+    --normalize ${normalize} \
+    --use_instruction ${use_instruction} \
+    --output_dir <output dir>
+```
+The "<>" should be replaced with your actual setting.<br>
+This is a general script that can be used to evaluate other huggingface embedding models, but you need to ensure that the pooling and other configurations are correct.
+
+### Sentence-transformers
+
+```
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("QZhou-Embedding")
+
+model = SentenceTransformer(
+    "QZhou-Embedding",
+    model_kwargs={"device_map": "auto", "trust_remote_code": True},
+    tokenizer_kwargs={"padding_side": "left", "trust_remote_code": True},
+    trust_remote_code=True
+)
+
+queries = [
+    "What is photosynthesis?",
+    "Who invented the telephone?",
+]
+documents = [
+    "Photosynthesis is the process by which green plants use sunlight, carbon dioxide, and water to produce glucose and oxygen. This biochemical reaction occurs in chloroplasts.",
+    "Alexander Graham Bell is credited with inventing the first practical telephone in 1876, receiving US patent number 174,465 for his device."
+]
+
+query_embeddings = model.encode(queries, prompt_name="query", normalize_embeddings=True)
+document_embeddings = model.encode(documents, normalize_embeddings=True)
+
+similarity = model.similarity(query_embeddings, document_embeddings)
+```
+
+### Huggingface Transformers
+
+```
+import torch
+import torch.nn.functional as F
+
+from torch import Tensor
+from transformers import AutoTokenizer, AutoModel
+
+
+def last_token_pool(last_hidden_states: Tensor,
+                 attention_mask: Tensor) -> Tensor:
+    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
+    if left_padding:
+        return last_hidden_states[:, -1]
+    else:
+        sequence_lengths = attention_mask.sum(dim=1) - 1
+        batch_size = last_hidden_states.shape[0]
+        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]
+
+
+def get_detailed_instruct(task_description: str, query: str) -> str:
+    return f'Instruct: {task_description}\nQuery:{query}'
+
+task = 'Given a web search query, retrieve relevant passages that answer the query'
+
+queries = [
+    get_detailed_instruct(task, 'What is photosynthesis?'),
+    get_detailed_instruct(task, 'Who invented the telephone?')
+]
+
+documents = [
+    "Photosynthesis is the process by which green plants use sunlight, carbon dioxide, and water to produce glucose and oxygen. This biochemical reaction occurs in chloroplasts.",
+    "Alexander Graham Bell is credited with inventing the first practical telephone in 1876, receiving US patent number 174,465 for his device."
+]
+
+input_texts = queries + documents
+
+tokenizer = AutoTokenizer.from_pretrained('QZhou-Embedding', padding_side='left', trust_remote_code=True)
+model = AutoModel.from_pretrained('QZhou-Embedding', trust_remote_code=True, device_map='auto')
+
+batch_dict = tokenizer(
+    input_texts,
+    padding=True,
+    truncation=True,
+    max_length=8192,
+    return_tensors="pt",
+)
+batch_dict.to(model.device)
+outputs = model(**batch_dict)
+embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
+
+embeddings = F.normalize(embeddings, p=2, dim=1)
+scores = (embeddings[:2] @ embeddings[2:].T)
+```

README_zh.md

@@ -0,0 +1,215 @@
+---
+license: apache-2.0
+tags:
+  - sentence-transformers
+  - sentence-similarity
+  - mteb
+  - retriever
+  - text-embeddings-inference
+---
+# QZhou-Embedding
+<div align="center">
+<img src="assets/image-1.png" width="800" height="300"></img>
+</div>
+
+## 简介
+我们发布<a href="https://huggingface.co/Kingsoft-LLM/QZhou-Embedding">QZhou-Embedding</a>(轻舟Embedding😈😈😈)，面向通用领域的文本向量表示大模型，擅长各种文本嵌入（检索、重排、句对相似度、分类）任务。得益于基础模型在海量文本上预训练获得的通用语言能力，QZhou-Embedding能够获得更加强大的文本嵌入表示。QZhou-Embedding使用百万量级高质量开源检索数据，以及500万+高质量合成数据（改写、扩展两大合成技术）进行持续训练。我们通过第一阶段检索训练为模型提供query-doc语义匹配能力基础，第二阶段的STS、聚类等多维度能力训练帮助模型在各种场景下持续突破。QZhou-Embedding的模型参数为7B，具备最大8k的长文本向量嵌入能力。在mteb/cmteb评测基准上取得均值全榜最高，各任务指标方面，聚类、句对分类、重排、STS任务指标均值全榜最高的效果。
+
+## QZhou-Embedding基本特点
+
+- 强大的文本嵌入能力；
+- 长上下文：最大支持8k；
+- 参数量7B
+
+
+## 技术介绍
+### 统一任务建模框架
+将文本嵌入目标统一为三大问题建模优化，提出统一的训练数据结构化方案和对应的训练机制---可融入大部分开源数据作为检索训练集，可结构化数据如下：
+- 检索
+  - title-body
+  - title-abstract
+  - 问答类数据
+  - 阅读理解
+  - ...
+
+- STS
+  - 文本对+{true, false}、{yes, no}标签
+  - 文本对+分数（如0.2、3.1、4.8等）
+  - NLI数据：文本对+{'entailment', 'neutral', 'contradiction'}标签
+
+- CLS
+  - 句子+类标签
+
+<div align="center"><img src="assets/image-18.png" width="1000" height="600"></img></div>
+<div align="center"><img src="assets/image-16.png" width="1000" height="550"></img></div>
+
+### 训练目标
+
+- 检索：使用InfoNCE对比学习loss函数，效仿gte/qwen3-embedding的改进增加q-q对负样例惩罚<br>
+$$
+L_{ret}=-\frac{1}{n}\sum_{i} log{\frac{e^{sim(q_i,d_i^+)/\tau}}{e^{sim(q_i,d_i^+)/\tau}+\sum_{j}e^{sim(q_i,d_j^-)/\tau}+\sum_{j≠i}e^{sim(q_i,q_j)/\tau}}}
+$$
+
+- STS：使用Cosent loss：
+$$
+L_{cosent}=log \bigg(1+\sum_{sim(i,j)>sim(k,l)}exp(\frac{sim(x_k, x_l)-sim(x_i,x_j)}{\tau})\bigg)
+$$
+
+- CLS：同检索一致使用InfoNCE loss，但In-Batch Negative时由于同类冲突概率大，使用mask机制掩盖不同样本共享的负样例中的同类样本。
+$$
+L_{ret}=-\frac{1}{n}\sum_{i} log{\frac{e^{sim(t_i,t_i^+)/\tau}}{e^{sim(t_i,t_i^+)/\tau}+\sum_{n}MASK(t_i,t_{i,n}^-)·e^{sim(t_i,t_{i,n}^-)/\tau}+\sum_{j≠i}MASK(t_i,t_j)·e^{sim(t_i,t_j)/\tau}+\sum_{j≠i}\sum_{n}MASK(t_i,t_{j,n}^-)e^{sim(t_i,t_{j,n}^-)/\tau}}}
+$$
+$$
+其中C_{t_i}=C_{t_i^+}
+$$
+$$
+MASK(t_i, t_j)=
+\begin{cases}
+0 & \quad \text{if } C_{t_i}=C_{t_j}, \\
+1 & \quad \text{otherwise}
+\end{cases}
+$$
+其中${C_{t_i}}$表示样本${t_i}$的类标签，n是单条数据的负样本数。
+
+### 特征增强数据合成技术
+在当今大模型语言及创作能力强大的背景下，我们充分利用了大模型API设计数据合成技术。针对训练集中存在数据少、话题狭隘等问题，我们提出改写、扩展合成技术；同时为增强训练时的负样例难度，我们在现有基于强大Embedding实现难负例采样的基础上，使用基于大模型的难负样例合成技术。几种技术介绍如下：
+<div align="center"><img src="assets/image-9.png" width="930" height="290"></img></div>
+<div align="center"><img src="assets/image-10.png" width="880" height="220"></img></div>
+<div align="center"><img src="assets/image-11.png" width="880" height="210"></img></div>
+
+想要获取更多信息（如评测脚本、指令格式等），欢迎访问我们的Github：<a href="https://github.com/Kingsoft-LLM/QZhou-Embedding">GitHub</a>
+
+## 评测结果
+### mteb榜单明细
+<div align="center"><img src="assets/image-7.png" width="1100" height="260"></img></div>
+
+### cmteb榜单明细
+<div align="center"><img src="assets/image-8.png" width="1000" height="260"></img></div>
+
+## 使用指南
+### 完全复现榜单结果
+我们提供详细的参数、环境配置，以便能够在自己的机器上完全跑出跟榜单一致的结果，包括环境依赖、模型参数等配置。
+#### 环境依赖版本
+- Python: 3.10.12
+- Sentence Transformers: 3.4.1
+- Transformers: 4.51.1
+- PyTorch: 2.7.1
+- Accelerate: 1.3.0
+- Datasets: 3.2.0
+- Tokenizers: 0.21.2
+#### 模型加载参数
+torch_dtype=torch.bfloat16<br>
+attn_implementation='sdpa'<br>
+**注：** 榜单结果使用了sdpa模式，其他模式('eager'、 'flash_attention_2')存在偏差，但不影响整体表现
+#### 指令添加规则
+在我们的<a href="https://github.com/Kingsoft-LLM/QZhou-Embedding">GitHub</a>上可以找到。
+#### 评测代码使用
+在<a href="https://github.com/Kingsoft-LLM/QZhou-Embedding">GitHub</a>上找到我们的评测代码，其中mteb评测脚本是**run_mteb_all_v2.py**，cmteb评测脚本是**run_cmteb_all.py**，运行如下命令：
+```
+POOLING_MODE=mean
+normalize=true
+use_instruction=true
+export TOKENIZERS_PARALLELISM=true
+
+model_name_or_path=模型目录位置
+
+python3 ./run_cmteb_all.py \
+    --model_name_or_path ${model_name_or_path}  \
+    --pooling_mode ${POOLING_MODE} \
+    --normalize ${normalize} \
+    --use_instruction ${use_instruction} \
+    --output_dir 结果输出路径
+
+python3 ./run_mteb_all_v2.py \
+    --model_name_or_path ${model_name_or_path}  \
+    --pooling_mode ${POOLING_MODE} \
+    --normalize ${normalize} \
+    --use_instruction ${use_instruction} \
+    --output_dir 结果输出路径
+```
+这是一套通用脚本，可以用于其他huggingface embedding模型的评测，但需要确保pooling等配置正确。
+
+### Sentence Transformers
+
+```
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("QZhou-Embedding")
+
+model = SentenceTransformer(
+    "QZhou-Embedding",
+    model_kwargs={"device_map": "auto", "trust_remote_code": True},
+    tokenizer_kwargs={"padding_side": "left", "trust_remote_code": True},
+    trust_remote_code=True
+)
+
+queries = [
+    "What is photosynthesis?",
+    "Who invented the telephone?",
+]
+documents = [
+    "Photosynthesis is the process by which green plants use sunlight, carbon dioxide, and water to produce glucose and oxygen. This biochemical reaction occurs in chloroplasts.",
+    "Alexander Graham Bell is credited with inventing the first practical telephone in 1876, receiving US patent number 174,465 for his device."
+]
+
+query_embeddings = model.encode(queries, prompt_name="query", normalize_embeddings=True)
+document_embeddings = model.encode(documents, normalize_embeddings=True)
+
+similarity = model.similarity(query_embeddings, document_embeddings)
+```
+
+### Huggingface Transformers
+
+```
+import torch
+import torch.nn.functional as F
+
+from torch import Tensor
+from transformers import AutoTokenizer, AutoModel
+
+
+def last_token_pool(last_hidden_states: Tensor,
+                 attention_mask: Tensor) -> Tensor:
+    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
+    if left_padding:
+        return last_hidden_states[:, -1]
+    else:
+        sequence_lengths = attention_mask.sum(dim=1) - 1
+        batch_size = last_hidden_states.shape[0]
+        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]
+
+
+def get_detailed_instruct(task_description: str, query: str) -> str:
+    return f'Instruct: {task_description}\nQuery:{query}'
+
+task = 'Given a web search query, retrieve relevant passages that answer the query'
+
+queries = [
+    get_detailed_instruct(task, 'What is photosynthesis?'),
+    get_detailed_instruct(task, 'Who invented the telephone?')
+]
+
+documents = [
+    "Photosynthesis is the process by which green plants use sunlight, carbon dioxide, and water to produce glucose and oxygen. This biochemical reaction occurs in chloroplasts.",
+    "Alexander Graham Bell is credited with inventing the first practical telephone in 1876, receiving US patent number 174,465 for his device."
+]
+
+input_texts = queries + documents
+
+tokenizer = AutoTokenizer.from_pretrained('QZhou-Embedding', padding_side='left', trust_remote_code=True)
+model = AutoModel.from_pretrained('QZhou-Embedding', trust_remote_code=True, device_map='auto')
+
+batch_dict = tokenizer(
+    input_texts,
+    padding=True,
+    truncation=True,
+    max_length=8192,
+    return_tensors="pt",
+)
+batch_dict.to(model.device)
+outputs = model(**batch_dict)
+embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
+
+embeddings = F.normalize(embeddings, p=2, dim=1)
+scores = (embeddings[:2] @ embeddings[2:].T)
+```

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dae144628ac262a2e3e585e2a04a08f285efa689f41c1a71432cec49db080263
+size 4877660152

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:890cf237c6407600e7b86ef0a4645614ccebbff609b338e268379e662e18f039
+size 4932750280

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3184d6e94fe1e62efb858cfb4336a7ccfb6b2e2620333bda0c6ac62123c3726f
+size 4330864528