Link to paper
Browse filesThis PR ensures https://huggingface.co/papers/2502.07983 shows up at the top and makes sure the links are all on the same line.
README.md
CHANGED
|
@@ -12,7 +12,6 @@ tags:
|
|
| 12 |
- human
|
| 13 |
---
|
| 14 |
|
| 15 |
-
|
| 16 |
<font size=5>**MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Visual Conditioned Parallel Denoising**</font>
|
| 17 |
</br>
|
| 18 |
Zhiqiang Xia <sup>\*</sup>,
|
|
@@ -27,7 +26,7 @@ Wenjiang Zhou
|
|
| 27 |
</br>
|
| 28 |
Lyra Lab, Tencent Music Entertainment
|
| 29 |
|
| 30 |
-
**[github](https://github.com/TMElyralab/MuseV)
|
| 31 |
|
| 32 |
|
| 33 |
We have setup **the world simulator vision since March 2023, believing diffusion models can simulate the world**. `MuseV` was a milestone achieved around **July 2023**. Amazed by the progress of Sora, we decided to opensource `MuseV`, hopefully it will benefit the community. Next we will move on to the promising diffusion+transformer scheme.
|
|
@@ -182,7 +181,7 @@ Bellow Case could be found in `configs/tasks/example.yaml`
|
|
| 182 |
</td>
|
| 183 |
<td>
|
| 184 |
<video width="400" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/P_Y5jUO1EJ6n3Z4qd1xh1.mp4"></video>
|
| 185 |
-
</td>
|
| 186 |
<td>
|
| 187 |
(masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
|
| 188 |
soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
|
|
@@ -364,9 +363,9 @@ There are still many limitations, including
|
|
| 364 |
1. Lack of generalization ability. Some visual condition image perform well, some perform bad. Some t2i pretraied model perform well, some perform bad.
|
| 365 |
1. Limited types of video generation and limited motion range, partly because of limited types of training data. The released `MuseV` has been trained on approximately 60K human text-video pairs with resolution `512*320`. `MuseV` has greater motion range while lower video quality at lower resolution. `MuseV` tends to generate less motion range with high video quality. Trained on larger, higher resolution, higher quality text-video dataset may make `MuseV` better.
|
| 366 |
1. Watermarks may appear because of `webvid`. A cleaner dataset withour watermarks may solve this issue.
|
| 367 |
-
|
| 368 |
-
|
| 369 |
-
|
| 370 |
|
| 371 |
<!-- # Contribution 暂时不需要组织开源共建 -->
|
| 372 |
|
|
@@ -383,5 +382,5 @@ There are still many limitations, including
|
|
| 383 |
1. `code`: The code of MuseV is released under the MIT License. There is no limitation for both academic and commercial usage.
|
| 384 |
1. `model`: The trained model are available for non-commercial research purposes only.
|
| 385 |
1. `other opensource model`: Other open-source models used must comply with their license, such as `insightface`, `IP-Adapter`, `ft-mse-vae`, etc.
|
| 386 |
-
|
| 387 |
-
|
|
|
|
| 12 |
- human
|
| 13 |
---
|
| 14 |
|
|
|
|
| 15 |
<font size=5>**MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Visual Conditioned Parallel Denoising**</font>
|
| 16 |
</br>
|
| 17 |
Zhiqiang Xia <sup>\*</sup>,
|
|
|
|
| 26 |
</br>
|
| 27 |
Lyra Lab, Tencent Music Entertainment
|
| 28 |
|
| 29 |
+
**[github](https://github.com/TMElyralab/MuseV) [huggingface](https://huggingface.co/TMElyralab/MuseV) [HuggingfaceSpace](https://huggingface.co/spaces/AnchorFake/MuseVDemo) [project](comming soon) [technical report](https://huggingface.co/papers/2502.07983) (comming soon)**
|
| 30 |
|
| 31 |
|
| 32 |
We have setup **the world simulator vision since March 2023, believing diffusion models can simulate the world**. `MuseV` was a milestone achieved around **July 2023**. Amazed by the progress of Sora, we decided to opensource `MuseV`, hopefully it will benefit the community. Next we will move on to the promising diffusion+transformer scheme.
|
|
|
|
| 181 |
</td>
|
| 182 |
<td>
|
| 183 |
<video width="400" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/P_Y5jUO1EJ6n3Z4qd1xh1.mp4"></video>
|
| 184 |
+
</td>
|
| 185 |
<td>
|
| 186 |
(masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
|
| 187 |
soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
|
|
|
|
| 363 |
1. Lack of generalization ability. Some visual condition image perform well, some perform bad. Some t2i pretraied model perform well, some perform bad.
|
| 364 |
1. Limited types of video generation and limited motion range, partly because of limited types of training data. The released `MuseV` has been trained on approximately 60K human text-video pairs with resolution `512*320`. `MuseV` has greater motion range while lower video quality at lower resolution. `MuseV` tends to generate less motion range with high video quality. Trained on larger, higher resolution, higher quality text-video dataset may make `MuseV` better.
|
| 365 |
1. Watermarks may appear because of `webvid`. A cleaner dataset withour watermarks may solve this issue.
|
| 366 |
+
4. Limited types of long video generation. Visual Conditioned Parallel Denoise can solve accumulated error of video generation, but the current method is only suitable for relatively fixed camera scenes.
|
| 367 |
+
5. Undertrained referencenet and IP-Adapter, beacause of limited time and limited resources.
|
| 368 |
+
6. Understructured code. `MuseV` supports rich and dynamic features, but with complex and unrefacted codes. It takes time to familiarize.
|
| 369 |
|
| 370 |
<!-- # Contribution 暂时不需要组织开源共建 -->
|
| 371 |
|
|
|
|
| 382 |
1. `code`: The code of MuseV is released under the MIT License. There is no limitation for both academic and commercial usage.
|
| 383 |
1. `model`: The trained model are available for non-commercial research purposes only.
|
| 384 |
1. `other opensource model`: Other open-source models used must comply with their license, such as `insightface`, `IP-Adapter`, `ft-mse-vae`, etc.
|
| 385 |
+
4. The testdata are collected from internet, which are available for non-commercial research purposes only.
|
| 386 |
+
5. `AIGC`: This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.
|