Add pipeline tag, link to paper and code (#1)

- Add pipeline tag, link to paper and code (a3c7474a5a49721ab31666e07c5c55b4f366485e)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -1,87 +1,90 @@
----
-license: apache-2.0
-language:
-- en
-inference: false
----
-<h1>VPO: Aligning Text-to-Video Generation Models with Prompt Optimization</h1>
-
-- **Repository:** https://github.com/thu-coai/VPO
-<!-- - **Paper:**  -->
-- **Data:** https://huggingface.co/datasets/CCCCCC/VPO
-
-# VPO
-VPO is a principled prompt optimization framework grounded in the principles of harmlessness, accuracy, and helpfulness.
-VPO employs a two-stage process that first constructs a supervised fine-tuning dataset guided by safety and alignment, and then conducts preference learning with both text-level and video-level feedback. As a result, VPO preserves user intent while enhancing video quality and safety.
-
-## Model Details
-
-### Video Generation Model
-This model is trained to optimize user prompt for CogVideoX-5B. [VPO-2B](https://huggingface.co/CCCCCC/VPO-2B) is for CogVideoX-2B.
-
-### Data
-Our dataset can be found [here](https://huggingface.co/datasets/CCCCCC/VPO).
-
-### Language
-English
-
-## Intended Use
-
-### Prompt Template 
-We adopt a prompt template as 
-```
-In this task, your goal is to expand the user's short query into a detailed and well-structured English prompt for generating short videos.
-
-Please ensure that the generated video prompt adheres to the following principles:
-
-1. **Harmless**: The prompt must be safe, respectful, and free from any harmful, offensive, or unethical content.  
-2. **Aligned**: The prompt should fully preserve the user's intent, incorporating all relevant details from the original query while ensuring clarity and coherence.  
-3. **Helpful for High-Quality Video Generation**: The prompt should be descriptive and vivid to facilitate high-quality video creation. Keep the scene feasible and well-suited for a brief duration, avoiding unnecessary complexity or unrealistic elements not mentioned in the query.
-
-User Query:{user prompt}
-
-Video Prompt:
-```
-
-### Inference code
-Here is an example code for inference: 
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_path = ''
-
-prompt_template = """In this task, your goal is to expand the user's short query into a detailed and well-structured English prompt for generating short videos.
-
-Please ensure that the generated video prompt adheres to the following principles:
-
-1. **Harmless**: The prompt must be safe, respectful, and free from any harmful, offensive, or unethical content.  
-2. **Aligned**: The prompt should fully preserve the user's intent, incorporating all relevant details from the original query while ensuring clarity and coherence.  
-3. **Helpful for High-Quality Video Generation**: The prompt should be descriptive and vivid to facilitate high-quality video creation. Keep the scene feasible and well-suited for a brief duration, avoiding unnecessary complexity or unrealistic elements not mentioned in the query.
-
-User Query:{}
-
-Video Prompt:"""
-
-device = 'cuda:0'
-model = AutoModelForCausalLM.from_pretrained(model_path).half().eval().to(device)
-# for 8bit
-# model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device, load_in_8bit=True)
-tokenizer = AutoTokenizer.from_pretrained(model_path)
-
-text = "a cute dog on the grass"
-messgae = [{'role': 'user', 'content': prompt_template.format(text)}]
-
-model_inputs = tokenizer.apply_chat_template(messgae, add_generation_prompt=True, tokenize=True, return_tensors="pt").to(device)
-output = model.generate(model_inputs, max_new_tokens=1024, do_sample=True, top_p=1.0, temperature=0.7, num_beams=1)
-resp = tokenizer.decode(output[0]).split('<|start_header_id|>assistant<|end_header_id|>')[1].split('<|eot_id|>')[0].strip()
-
-print(resp)
-```
-See our [Github Repo](https://github.com/thu-coai/VPO) for more detailed usage (e.g. Inference with Vllm).
-
-
-<!-- ## Citation
-If you find our model is useful in your work, please cite it with:
-```
-
+---
+language:
+- en
+license: apache-2.0
+inference: false
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+<h1>VPO: Aligning Text-to-Video Generation Models with Prompt Optimization</h1>
+
+- **Repository:** https://github.com/thu-coai/VPO
+- **Paper:** [VPO: Aligning Text-to-Video Generation Models with Prompt Optimization](https://huggingface.co/papers/2503.20491)
+- **Data:** https://huggingface.co/datasets/CCCCCC/VPO
+
+# VPO
+VPO is a principled prompt optimization framework grounded in the principles of harmlessness, accuracy, and helpfulness.
+VPO employs a two-stage process that first constructs a supervised fine-tuning dataset guided by safety and alignment, and then conducts preference learning with both text-level and video-level feedback. As a result, VPO preserves user intent while enhancing video quality and safety.
+
+## Model Details
+
+### Video Generation Model
+This model is trained to optimize user prompt for CogVideoX-5B. [VPO-2B](https://huggingface.co/CCCCCC/VPO-2B) is for CogVideoX-2B.
+
+### Data
+Our dataset can be found [here](https://huggingface.co/datasets/CCCCCC/VPO).
+
+### Language
+English
+
+## Intended Use
+
+### Prompt Template 
+We adopt a prompt template as 
+```
+In this task, your goal is to expand the user's short query into a detailed and well-structured English prompt for generating short videos.
+
+Please ensure that the generated video prompt adheres to the following principles:
+
+1. **Harmless**: The prompt must be safe, respectful, and free from any harmful, offensive, or unethical content.  
+2. **Aligned**: The prompt should fully preserve the user's intent, incorporating all relevant details from the original query while ensuring clarity and coherence.  
+3. **Helpful for High-Quality Video Generation**: The prompt should be descriptive and vivid to facilitate high-quality video creation. Keep the scene feasible and well-suited for a brief duration, avoiding unnecessary complexity or unrealistic elements not mentioned in the query.
+
+User Query:{user prompt}
+
+Video Prompt:
+```
+
+### Inference code
+Here is an example code for inference: 
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_path = ''
+
+prompt_template = """In this task, your goal is to expand the user's short query into a detailed and well-structured English prompt for generating short videos.
+
+Please ensure that the generated video prompt adheres to the following principles:
+
+1. **Harmless**: The prompt must be safe, respectful, and free from any harmful, offensive, or unethical content.  
+2. **Aligned**: The prompt should fully preserve the user's intent, incorporating all relevant details from the original query while ensuring clarity and coherence.  
+3. **Helpful for High-Quality Video Generation**: The prompt should be descriptive and vivid to facilitate high-quality video creation. Keep the scene feasible and well-suited for a brief duration, avoiding unnecessary complexity or unrealistic elements not mentioned in the query.
+
+User Query:{}
+
+Video Prompt:"""
+
+device = 'cuda:0'
+model = AutoModelForCausalLM.from_pretrained(model_path).half().eval().to(device)
+# for 8bit
+# model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device, load_in_8bit=True)
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+
+text = "a cute dog on the grass"
+messgae = [{'role': 'user', 'content': prompt_template.format(text)}]
+
+model_inputs = tokenizer.apply_chat_template(messgae, add_generation_prompt=True, tokenize=True, return_tensors="pt").to(device)
+output = model.generate(model_inputs, max_new_tokens=1024, do_sample=True, top_p=1.0, temperature=0.7, num_beams=1)
+resp = tokenizer.decode(output[0]).split('<|start_header_id|>assistant<|end_header_id|>')[1].split('<|eot_id|>')[0].strip()
+
+print(resp)
+```
+See our [Github Repo](https://github.com/thu-coai/VPO) for more detailed usage (e.g. Inference with Vllm).
+
+
+<!-- ## Citation
+If you find our model is useful in your work, please cite it with:
+```
+
 ``` -->