Update README.md
Browse files
README.md
CHANGED
|
@@ -91,32 +91,6 @@ All videos are available in this [Link](https://cloudbook-public-daily.oss-cn-ha
|
|
| 91 |
- [x] Release diffusers version and optimize the GPU memory usage
|
| 92 |
- [x] Release complete version of Tora
|
| 93 |
|
| 94 |
-
## 🧨 Diffusers verision
|
| 95 |
-
|
| 96 |
-
Please refer to [the diffusers version](diffusers-version/README.md) for details.
|
| 97 |
-
|
| 98 |
-
## 🐍 Installation
|
| 99 |
-
|
| 100 |
-
Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.
|
| 101 |
-
|
| 102 |
-
```bash
|
| 103 |
-
# Clone this repository.
|
| 104 |
-
git clone https://github.com/alibaba/Tora.git
|
| 105 |
-
cd Tora
|
| 106 |
-
|
| 107 |
-
# Install Pytorch (we use Pytorch 2.4.0) and torchvision following the official instructions: https://pytorch.org/get-started/previous-versions/. For example:
|
| 108 |
-
conda create -n tora python==3.10
|
| 109 |
-
conda activate tora
|
| 110 |
-
conda install pytorch==2.4.0 torchvision==0.19.0 pytorch-cuda=12.1 -c pytorch -c nvidia
|
| 111 |
-
|
| 112 |
-
# Install requirements
|
| 113 |
-
cd modules/SwissArmyTransformer
|
| 114 |
-
pip install -e .
|
| 115 |
-
cd ../../sat
|
| 116 |
-
pip install -r requirements.txt
|
| 117 |
-
cd ..
|
| 118 |
-
```
|
| 119 |
-
|
| 120 |
## 📦 Model Weights
|
| 121 |
|
| 122 |
### Folder Structure
|
|
@@ -182,91 +156,6 @@ git clone https://www.modelscope.cn/xiaoche/Tora.git
|
|
| 182 |
- T5: [text_encoder](https://huggingface.co/THUDM/CogVideoX-2b/tree/main/text_encoder), [tokenizer](https://huggingface.co/THUDM/CogVideoX-2b/tree/main/tokenizer)
|
| 183 |
- Tora t2v model weights: [Link](https://cloudbook-public-daily.oss-cn-hangzhou.aliyuncs.com/Tora_t2v/mp_rank_00_model_states.pt). Downloading this weight requires following the [CogVideoX License](CogVideoX_LICENSE).
|
| 184 |
|
| 185 |
-
## 🔄 Inference
|
| 186 |
-
|
| 187 |
-
### Text to Video
|
| 188 |
-
It requires around 30 GiB GPU memory tested on NVIDIA A100.
|
| 189 |
-
|
| 190 |
-
```bash
|
| 191 |
-
cd sat
|
| 192 |
-
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU sample_video.py --base configs/tora/model/cogvideox_5b_tora.yaml configs/tora/inference_sparse.yaml --load ckpts/tora/t2v --output-dir samples --point_path trajs/coaster.txt --input-file assets/text/t2v/examples.txt
|
| 193 |
-
```
|
| 194 |
-
|
| 195 |
-
You can change the `--input-file` and `--point_path` to your own prompts and trajectory points files. Please note that the trajectory is drawn on a 256x256 canvas.
|
| 196 |
-
|
| 197 |
-
Replace `$N_GPU` with the number of GPUs you want to use.
|
| 198 |
-
|
| 199 |
-
### Image to Video
|
| 200 |
-
|
| 201 |
-
```bash
|
| 202 |
-
cd sat
|
| 203 |
-
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU sample_video.py --base configs/tora/model/cogvideox_5b_tora_i2v.yaml configs/tora/inference_sparse.yaml --load ckpts/tora/i2v --output-dir samples --point_path trajs/sawtooth.txt --input-file assets/text/i2v/examples.txt --img_dir assets/images --image2video
|
| 204 |
-
```
|
| 205 |
-
|
| 206 |
-
The first frame images should be placed in the `--img_dir`. The names of these images should be specified in the corresponding text prompt in `--input-file`, seperated by `@@`.
|
| 207 |
-
|
| 208 |
-
### Recommendations for Text Prompts
|
| 209 |
-
|
| 210 |
-
For text prompts, we highly recommend using GPT-4 to enhance the details. Simple prompts may negatively impact both visual quality and motion control effectiveness.
|
| 211 |
-
|
| 212 |
-
You can refer to the following resources for guidance:
|
| 213 |
-
|
| 214 |
-
- [CogVideoX Documentation](https://github.com/THUDM/CogVideo/blob/main/inference/convert_demo.py)
|
| 215 |
-
- [OpenSora Scripts](https://github.com/hpcaitech/Open-Sora/blob/main/scripts/inference.py)
|
| 216 |
-
|
| 217 |
-
## 🖥️ Gradio Demo
|
| 218 |
-
|
| 219 |
-
Usage:
|
| 220 |
-
|
| 221 |
-
```bash
|
| 222 |
-
cd sat
|
| 223 |
-
python app.py --load ckpts/tora/t2v
|
| 224 |
-
```
|
| 225 |
-
|
| 226 |
-
## 🧠 Training
|
| 227 |
-
|
| 228 |
-
### Data Preparation
|
| 229 |
-
|
| 230 |
-
Following this guide https://github.com/THUDM/CogVideo/blob/main/sat/README.md#preparing-the-dataset, structure the datasets as follows:
|
| 231 |
-
|
| 232 |
-
```
|
| 233 |
-
.
|
| 234 |
-
├── labels
|
| 235 |
-
│ ├── 1.txt
|
| 236 |
-
│ ├── 2.txt
|
| 237 |
-
│ ├── ...
|
| 238 |
-
└── videos
|
| 239 |
-
├── 1.mp4
|
| 240 |
-
├── 2.mp4
|
| 241 |
-
├── ...
|
| 242 |
-
```
|
| 243 |
-
|
| 244 |
-
Training data examples are in `sat/training_examples`
|
| 245 |
-
|
| 246 |
-
### Text to Video
|
| 247 |
-
|
| 248 |
-
It requires around 60 GiB GPU memory tested on NVIDIA A100.
|
| 249 |
-
|
| 250 |
-
Replace `$N_GPU` with the number of GPUs you want to use.
|
| 251 |
-
|
| 252 |
-
- Stage 1
|
| 253 |
-
|
| 254 |
-
```bash
|
| 255 |
-
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU train_video.py --base configs/tora/model/cogvideox_5b_tora.yaml configs/tora/train_dense.yaml --experiment-name "t2v-stage1"
|
| 256 |
-
```
|
| 257 |
-
|
| 258 |
-
- Stage 2
|
| 259 |
-
|
| 260 |
-
```bash
|
| 261 |
-
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True torchrun --standalone --nproc_per_node=$N_GPU train_video.py --base configs/tora/model/cogvideox_5b_tora.yaml configs/tora/train_sparse.yaml --experiment-name "t2v-stage2"
|
| 262 |
-
```
|
| 263 |
-
|
| 264 |
-
## 🎯 Troubleshooting
|
| 265 |
-
|
| 266 |
-
### 1. ValueError: Non-consecutive added token...
|
| 267 |
-
|
| 268 |
-
Upgrade the transformers package to 4.44.2. See [this](https://github.com/THUDM/CogVideo/issues/213) issue.
|
| 269 |
-
|
| 270 |
## 🤝 Acknowledgements
|
| 271 |
|
| 272 |
We would like to express our gratitude to the following open-source projects that have been instrumental in the development of our project:
|
|
|
|
| 91 |
- [x] Release diffusers version and optimize the GPU memory usage
|
| 92 |
- [x] Release complete version of Tora
|
| 93 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
## 📦 Model Weights
|
| 95 |
|
| 96 |
### Folder Structure
|
|
|
|
| 156 |
- T5: [text_encoder](https://huggingface.co/THUDM/CogVideoX-2b/tree/main/text_encoder), [tokenizer](https://huggingface.co/THUDM/CogVideoX-2b/tree/main/tokenizer)
|
| 157 |
- Tora t2v model weights: [Link](https://cloudbook-public-daily.oss-cn-hangzhou.aliyuncs.com/Tora_t2v/mp_rank_00_model_states.pt). Downloading this weight requires following the [CogVideoX License](CogVideoX_LICENSE).
|
| 158 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
## 🤝 Acknowledgements
|
| 160 |
|
| 161 |
We would like to express our gratitude to the following open-source projects that have been instrumental in the development of our project:
|