Improve model card: Add pipeline tag, library name, tags, project page, and sample usage
Browse filesThis PR significantly enhances the model card for the ReasonRank model by:
- Adding `pipeline_tag: text-ranking` to ensure discoverability under relevant tasks on the Hugging Face Hub.
- Specifying `library_name: transformers` to enable the "How to use" widget and indicate compatibility with the 🤗 Transformers library.
- Including relevant `tags` such as `qwen`, `reranker`, and `passage-ranking` for improved searchability.
- Integrating comprehensive details from the original GitHub repository, including a model overview, training data specifics, and performance results.
- Providing a practical Python code snippet for `Sample Usage` to help users quickly get started with inference.
- Ensuring links to the project page and original paper are present and consistent.
|
@@ -1,21 +1,219 @@
|
|
| 1 |
---
|
| 2 |
-
|
|
|
|
| 3 |
datasets:
|
| 4 |
- liuwenhan/reasonrank_data_sft
|
| 5 |
- liuwenhan/reasonrank_data_rl
|
| 6 |
- liuwenhan/reasonrank_data_13k
|
| 7 |
language:
|
| 8 |
- en
|
| 9 |
-
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
-
## Model Performance
|
| 17 |
<p align="center">
|
| 18 |
-
<img width="
|
| 19 |
</p>
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model:
|
| 3 |
+
- Qwen/Qwen2.5-7B-Instruct
|
| 4 |
datasets:
|
| 5 |
- liuwenhan/reasonrank_data_sft
|
| 6 |
- liuwenhan/reasonrank_data_rl
|
| 7 |
- liuwenhan/reasonrank_data_13k
|
| 8 |
language:
|
| 9 |
- en
|
| 10 |
+
license: mit
|
| 11 |
+
pipeline_tag: text-ranking
|
| 12 |
+
library_name: transformers
|
| 13 |
+
tags:
|
| 14 |
+
- qwen
|
| 15 |
+
- reranker
|
| 16 |
+
- passage-ranking
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability
|
| 20 |
+
|
| 21 |
+
<p align="center">
|
| 22 |
+
<a href="https://arxiv.org/pdf/2508.07050" target="_blank"><img src="https://img.shields.io/badge/Paper-arXiv-b5212f.svg?logo=arxiv"></a>
|
| 23 |
+
<a href="https://github.com/8421BCD/ReasonRank" target="_blank"><img src="https://img.shields.io/badge/GitHub-Repo-181717.svg?logo=github"></a>
|
| 24 |
+
<a href="https://brightbenchmark.github.io/" target="_blank"><img src="https://img.shields.io/badge/Project%20Page-BRIGHT-blue.svg"></a>
|
| 25 |
+
<a href="https://opensource.org/licenses/MIT"><img alt="License" src="https://img.shields.io/badge/LICENSE-MIT-green.svg"></a>
|
| 26 |
+
</p>
|
| 27 |
+
|
| 28 |
+
<p align="center">
|
| 29 |
+
🤗 <a href="https://huggingface.co/liuwenhan/reasonrank-7B" target="_blank">reasonrank-7B</a> |
|
| 30 |
+
🤗 <a href="https://huggingface.co/liuwenhan/reasonrank-32B" target="_blank">reasonrank-32B</a>
|
| 31 |
+
</p>
|
| 32 |
+
<p align="center">
|
| 33 |
+
🤗 <a href="https://huggingface.co/datasets/liuwenhan/reasonrank_data_13k" target="_blank">reasonrank_data_13k</a> |
|
| 34 |
+
🤗 <a href="https://huggingface.co/datasets/liuwenhan/reasonrank_data_sft" target="_blank">reasonrank_data_sft</a> |
|
| 35 |
+
🤗 <a href="https://huggingface.co/datasets/liuwenhan/reasonrank_data_rl" target="_blank">reasonrank_data_rl</a>
|
| 36 |
+
</p>
|
| 37 |
+
<h5 align="center"> If you like our project, please give us a star ⭐ on GitHub.</h5>
|
| 38 |
+
|
| 39 |
+
## 📣 Latest News
|
| 40 |
+
- **[Aug 9, 2025]**: 🏆 Our ReasonRank (32B) has achieved **SOTA performance 40.8** on **[BRIGHT leaderboard](https://brightbenchmark.github.io/)**!
|
| 41 |
+
- **[Aug 9, 2025]**: 📄 We uploaded our paper to the **[arXiv](https://arxiv.org/pdf/2508.07050)** and **[Hugging Face](https://huggingface.co/papers/2508.07050)**.
|
| 42 |
+
- **[Aug 9, 2025]**: 🔥 We released our **[🤗full reasonrank training data (13k)](https://huggingface.co/datasets/liuwenhan/reasonrank_data_13k)**, **[🤗cold-start SFT data](https://huggingface.co/datasets/liuwenhan/reasonrank_data_sft)** and **[🤗RL data](https://huggingface.co/datasets/liuwenhan/reasonrank_data_rl)**.
|
| 43 |
+
- **[Aug 9, 2025]**: 🔥 We released our reasoning-intensive reranker **[🤗reasonrank-7B](https://huggingface.co/liuwenhan/reasonrank-7B)** and **[🤗reasonrank-32B](https://huggingface.co/liuwenhan/reasonrank-32B)**.
|
| 44 |
+
- **[Aug 9, 2025]**: 🚀 We released our full codebase, including inference, SFT training, and RL training.
|
| 45 |
+
|
| 46 |
+
## 1. ReasonRank
|
| 47 |
+
|
| 48 |
+
### 💡 1.1 Overview
|
| 49 |
+
|
| 50 |
+
**ReasonRank** is a **reasoning-intensive passage reranker** tailored for reasoning-intensive ranking tasks. To train it, we first design an automated reasoning-intensive training data synthesis framework and synthesize 1.3k high-quality training data.
|
| 51 |
+
|
| 52 |
+
<p align="center">
|
| 53 |
+
<img width="80%" alt="image" src="https://8421bcd.oss-cn-beijing.aliyuncs.com/img/image-20250809002302377.png" />
|
| 54 |
+
</p>
|
| 55 |
+
|
| 56 |
+
Based on the training data, we design a two-stage training approach including **cold-start SFT** and **multi-view ranking reward RL** to inject listwise ranking ability to our ReasonRank.
|
| 57 |
+
|
| 58 |
+
<p align="center">
|
| 59 |
+
<img width="80%" alt="image" src="https://8421bcd.oss-cn-beijing.aliyuncs.com/img/image-20250809002546838.png" />
|
| 60 |
+
</p>
|
| 61 |
+
|
| 62 |
+
### 📊 1.2 Overall Performance
|
| 63 |
+
|
| 64 |
+
When using ReasonIR as initial passage retriever, our ReasonRank demonstrates strong overall ranking performance on BRIGHT benchmark, while showing superior efficiency compared with pointwise reasoning-intensive reranker Rank1.
|
| 65 |
|
|
|
|
| 66 |
<p align="center">
|
| 67 |
+
<img width="50%" alt="image" src="https://8421bcd.oss-cn-beijing.aliyuncs.com/img/image-20250809003636871.png" />
|
| 68 |
</p>
|
| 69 |
|
| 70 |
+
Besides, when using a higher-quality retrieval results (RaDeR + BM25 hybrid, provided by [RaDeR](https://github.com/Debrup-61/RaDeR/blob/main/BRIGHT_score_files/RaDeR-gte-Qwen2-LLMq_CoT_lexical/aops/hybrid_BM25_Rader.json)), our ReasonRank (32B) achieves SOTA performance **40.8** on [BRIGHT leaderboard](https://brightbenchmark.github.io/).
|
| 71 |
+
|
| 72 |
+
## 📂 2. The Introduction of ReasonRank Training Data
|
| 73 |
+
|
| 74 |
+
An important contribution of our work is our reasoning-intensive training data ([reasonrank_data_13k](https://huggingface.co/datasets/liuwenhan/reasonrank_data_13k)). The dataset fields of ``training_data_all.jsonl`` are as follows:
|
| 75 |
+
|
| 76 |
+
#### **Dataset Fields & Descriptions**
|
| 77 |
+
|
| 78 |
+
1. **`dataset`** *(str)*
|
| 79 |
+
- The dataset name of each piece of data (e.g., `"math-qa"`).
|
| 80 |
+
2. **`qid`** *(str)*
|
| 81 |
+
- The query ID. The content is provided in ``id_query/`` directory.
|
| 82 |
+
3. **`initial_list`** *(List[str])*
|
| 83 |
+
- The initial list of passage IDs before DeepSeek-R1 reranking. The content of each passage ID is provided in ``id_doc/`` directory.
|
| 84 |
+
4. **`final_list`** *(List[str])*
|
| 85 |
+
- The re-ranked list of passage IDs after listwisely reranking with DeepSeek-R1.
|
| 86 |
+
- Reflects the improved ranking based on reasoning-enhanced relevance scoring.
|
| 87 |
+
5. **`reasoning`** *(str)*
|
| 88 |
+
- A **step-by-step reasoning chain** outputted by DeepSeek-R1 while performing the listwise reranking.
|
| 89 |
+
6. **`relevant_docids`** *(List[str])*
|
| 90 |
+
- The ids of relevant passages in ``initial_list`` mined by DeepSeek-R1. The remaining passage ids in ``initial_list`` are irrelevant ones.
|
| 91 |
+
- Note that **`relevant_docids`** are not necessarily ranked at the top of **`final_list`** by the DeepSeek-R1, which may stem from inconsistencies in DeepSeek-R1’s judgments. To address this, you can apply the **self-consistency data filtering** technique proposed in our paper to select higher-quality data.
|
| 92 |
+
|
| 93 |
+
The statistics of dataset is shown in the figure below:
|
| 94 |
+
<p align="center">
|
| 95 |
+
<img width="80%" alt="image" src="https://github.com/user-attachments/assets/c04b9d1a-2f21-46f1-b23d-ad1f50d22fb8" />
|
| 96 |
+
</p>
|
| 97 |
+
|
| 98 |
+
#### **Example Entry**
|
| 99 |
+
|
| 100 |
+
```json
|
| 101 |
+
{
|
| 102 |
+
"dataset": "math-qa",
|
| 103 |
+
"qid": "math_1001",
|
| 104 |
+
"initial_list": ["math_test_intermediate_algebra_808", "math_train_intermediate_algebra_1471", ...],
|
| 105 |
+
"final_list": ["math_test_intermediate_algebra_808", "math_test_intermediate_algebra_1678", ...],
|
| 106 |
+
"reasoning": "Okay, I need to rank the 20 passages based on their relevance...",
|
| 107 |
+
"relevant_docids": ["math_test_intermediate_algebra_808", "math_train_intermediate_algebra_1471", "math_train_intermediate_algebra_993"]
|
| 108 |
+
}
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
#### **Application**
|
| 112 |
+
|
| 113 |
+
1. Training passage reranker: Given the reranked passage list, one can use our data to train a listwise reranker
|
| 114 |
+
2. Training passage retriever: Using the **`relevant_docids`** and the remaining irrelevant ids, one can train a passage retriever.
|
| 115 |
+
|
| 116 |
+
## ⚡ 3. Quick Start
|
| 117 |
+
|
| 118 |
+
This section provides a general guide on how to use ReasonRank for inference. For detailed environment setup, specific inference commands (including usage with ReasonIR or custom retrieval results), and in-depth training procedures (Cold-Start SFT, Multi-reward ranking RL), please refer to the [official GitHub repository](https://github.com/8421BCD/ReasonRank).
|
| 119 |
+
|
| 120 |
+
## Sample Usage
|
| 121 |
+
|
| 122 |
+
This model can be loaded and used with the `transformers` library. Below is a basic example demonstrating how to use the model for passage re-ranking. The model expects a specific chat-like format for input, including a system prompt and a user query with listed passages.
|
| 123 |
+
|
| 124 |
+
```python
|
| 125 |
+
import torch
|
| 126 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 127 |
+
|
| 128 |
+
model_name = "liuwenhan/reasonrank-7B" # Or "liuwenhan/reasonrank-32B"
|
| 129 |
+
|
| 130 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 131 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 132 |
+
model_name,
|
| 133 |
+
torch_dtype=torch.bfloat16,
|
| 134 |
+
device_map="auto",
|
| 135 |
+
trust_remote_code=True
|
| 136 |
+
).eval()
|
| 137 |
+
|
| 138 |
+
# System prompt as used in the paper for reasoning-intensive ranking
|
| 139 |
+
system_prompt = (
|
| 140 |
+
"You are a helpful and harmless AI assistant. You will be provided with a search query and a list of passages, "
|
| 141 |
+
"and your task is to re-rank the passages based on their relevance to the query. "
|
| 142 |
+
"You should follow a chain of thought to determine the most relevant passages. "
|
| 143 |
+
"Your final answer should be a list of the re-ranked passage numbers, separated by commas. "
|
| 144 |
+
"Do not include any other information or explanation in your final answer."
|
| 145 |
+
)
|
| 146 |
+
|
| 147 |
+
query = "What is the capital of France?"
|
| 148 |
+
passages = [
|
| 149 |
+
"Paris is the capital and most populous city of France.",
|
| 150 |
+
"The Eiffel Tower is a famous landmark in Paris.",
|
| 151 |
+
"France is a country located in Western Europe.",
|
| 152 |
+
"London is the capital of the United Kingdom."
|
| 153 |
+
]
|
| 154 |
+
|
| 155 |
+
# Construct the user message with query and passages
|
| 156 |
+
user_content = f"Search Query: {query}
|
| 157 |
+
"
|
| 158 |
+
for i, passage in enumerate(passages):
|
| 159 |
+
user_content += f"[{i+1}] {passage}
|
| 160 |
+
"
|
| 161 |
+
user_content += "Please re-rank the passages based on their relevance to the query. Provide a chain of thought and then the final re-ranked list."
|
| 162 |
+
|
| 163 |
+
messages = [
|
| 164 |
+
{"role": "system", "content": system_prompt},
|
| 165 |
+
{"role": "user", "content": user_content}
|
| 166 |
+
]
|
| 167 |
+
|
| 168 |
+
# Apply chat template
|
| 169 |
+
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
| 170 |
+
|
| 171 |
+
# Tokenize input
|
| 172 |
+
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device)
|
| 173 |
+
|
| 174 |
+
# Generate response
|
| 175 |
+
output_ids = model.generate(
|
| 176 |
+
input_ids,
|
| 177 |
+
max_new_tokens=256, # Adjust as needed for reasoning length
|
| 178 |
+
do_sample=False, # Typically deterministic for ranking/reasoning
|
| 179 |
+
temperature=0.1, # Low temperature for focused output
|
| 180 |
+
repetition_penalty=1.05,
|
| 181 |
+
eos_token_id=tokenizer.eos_token_id
|
| 182 |
+
)
|
| 183 |
+
|
| 184 |
+
# Decode output
|
| 185 |
+
generated_text = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
|
| 186 |
+
print(generated_text)
|
| 187 |
+
```
|
| 188 |
+
|
| 189 |
+
## Citation
|
| 190 |
+
|
| 191 |
+
If you find this work helpful, please cite our papers:
|
| 192 |
+
|
| 193 |
+
```bibtex
|
| 194 |
+
@misc{liu2025reasonrankempoweringpassageranking,
|
| 195 |
+
title={ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability},
|
| 196 |
+
author={Wenhan Liu and Xinyu Ma and Weiwei Sun and Yutao Zhu and Yuchen Li and Dawei Yin and Zhicheng Dou},
|
| 197 |
+
year={2025},
|
| 198 |
+
eprint={2508.07050},
|
| 199 |
+
archivePrefix={arXiv},
|
| 200 |
+
primaryClass={cs.IR},
|
| 201 |
+
url={https://arxiv.org/abs/2508.07050},
|
| 202 |
+
}
|
| 203 |
+
```
|
| 204 |
+
|
| 205 |
+
## 🤝 Acknowledge
|
| 206 |
+
|
| 207 |
+
The inference codes and training implementation build upon [RankLLM](https://github.com/castorini/rank_llm), [Llama Factory](https://github.com/hiyouga/LLaMA-Factory) and [verl](https://github.com/volcengine/verl). Our work is based on the [Qwen2.5](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) model series, and we sincerely thank the Qwen team for their outstanding contributions to the open-source community.
|
| 208 |
+
|
| 209 |
+
## 📄 License
|
| 210 |
+
|
| 211 |
+
This project is released under the [MIT License](LICENSE).
|
| 212 |
+
|
| 213 |
+
## 📞 Contact
|
| 214 |
+
|
| 215 |
+
For any questions or feedback, please reach out to us at [[email protected]]([email protected]).
|
| 216 |
+
|
| 217 |
+
## Star History
|
| 218 |
+
|
| 219 |
+
[](https://www.star-history.com/#8421bcd/reasonrank&Date)
|