docs/VACE.md

# VACE ControlNet Guide

VACE is a powerful ControlNet that enables Video-to-Video and Reference-to-Video generation. It allows you to inject your own images into output videos, animate characters, perform inpainting/outpainting, and continue videos.

## Overview

VACE is probably one of the most powerful Wan models available. With it, you can:
- Inject people or objects into scenes
- Animate characters
- Perform video inpainting and outpainting
- Continue existing videos
- Transfer motion from one video to another
- Change the style of scenes while preserving depth

## Getting Started

### Model Selection
1. Select either "Vace 1.3B" or "Vace 13B" from the dropdown menu
2. Note: VACE works best with videos up to 7 seconds with the Riflex option enabled

### Input Types

VACE accepts three types of visual hints (which can be combined):

#### 1. Control Video
- Transfer motion or depth to a new video
- Use only the first n frames and extrapolate the rest
- Perform inpainting with grey color (127) as mask areas
- Grey areas will be filled based on text prompt and reference images

#### 2. Reference Images
- Use as background/setting for the video
- Inject people or objects of your choice
- Select multiple reference images
- **Tip**: Replace complex backgrounds with white for better object integration
- Always describe injected objects/people explicitly in your text prompt

#### 3. Video Mask
- Stronger control over which parts to keep (black) or replace (white)
- Perfect for inpainting/outpainting
- Example: White mask except at beginning/end (black) keeps first/last frames while generating middle content

## Common Use Cases

### Motion Transfer
**Goal**: Animate a character of your choice using motion from another video
**Setup**: 
- Reference Images: Your character
- Control Video: Person performing desired motion
- Text Prompt: Describe your character and the action

### Object/Person Injection
**Goal**: Insert people or objects into a scene
**Setup**:
- Reference Images: The people/objects to inject
- Text Prompt: Describe the scene and explicitly mention the injected elements

### Character Animation
**Goal**: Animate a character based on text description
**Setup**:
- Control Video: Video of person moving
- Text Prompt: Detailed description of your character

### Style Transfer with Depth
**Goal**: Change scene style while preserving spatial relationships
**Setup**:
- Control Video: Original video (for depth information)
- Text Prompt: New style description

## Integrated Matanyone Tool

WanGP includes the Matanyone tool, specifically tuned for VACE workflows. This helps create control videos and masks simultaneously.

### Creating Face Replacement Masks
1. Load your video in Matanyone
2. Click on the face in the first frame
3. Create a mask for the face
4. Generate both control video and mask video with "Generate Video Matting"
5. Export to VACE with "Export to current Video Input and Video Mask"
6. Load replacement face image in Reference Images field

### Advanced Matanyone Tips
- **Negative Point Prompts**: Remove parts from current selection
- **Sub Masks**: Create multiple independent masks, then combine them
- **Background Masks**: Select everything except the character (useful for background replacement)
- Enable/disable sub masks in Matanyone settings

## Recommended Settings

### Quality Settings
- **Skip Layer Guidance**: Turn ON with default configuration for better results
- **Long Prompts**: Use detailed descriptions, especially for background elements not in reference images
- **Steps**: Use at least 15 steps for good quality, 30+ for best results

### Sliding Window Settings
For very long videos, configure sliding windows properly:

- **Window Size**: Set appropriate duration for your content
- **Overlap Frames**: Long enough for motion continuity, short enough to avoid blur propagation
- **Discard Last Frames**: Remove at least 4 frames from each window (VACE 1.3B tends to blur final frames)

### Background Removal
VACE includes automatic background removal options:
- Use for reference images containing people/objects
- **Don't use** for landscape/setting reference images (first reference image)
- Multiple background removal types available

## Window Sliding for Long Videos

Generate videos up to 1 minute by merging multiple windows:

### How It Works
- Each window uses corresponding time segment from control video
- Example: 0-4s control video → first window, 4-8s → second window, etc.
- Automatic overlap management ensures smooth transitions

### Settings
- **Window Size**: Duration of each generation window
- **Overlap Frames**: Frames shared between windows for continuity
- **Discard Last Frames**: Remove poor-quality ending frames
- **Add Overlapped Noise**: Reduce quality degradation over time

### Formula
```
Generated Frames = [Windows - 1] × [Window Size - Overlap - Discard] + Window Size
```

### Multi-Line Prompts (Experimental)
- Each line of prompt used for different window
- If more windows than prompt lines, last line repeats
- Separate lines with carriage return

## Advanced Features

### Extend Video
Click "Extend the Video Sample, Please!" during generation to add more windows dynamically.

### Noise Addition
Add noise to overlapped frames to hide accumulated errors and quality degradation.

### Frame Truncation
Automatically remove lower-quality final frames from each window (recommended: 4 frames for VACE 1.3B).

## External Resources

### Official VACE Resources
- **GitHub**: https://github.com/ali-vilab/VACE/tree/main/vace/gradios
- **User Guide**: https://github.com/ali-vilab/VACE/blob/main/UserGuide.md
- **Preprocessors**: Gradio tools for preparing materials

### Recommended External Tools
- **Annotation Tools**: For creating precise masks
- **Video Editors**: For preparing control videos
- **Background Removal**: For cleaning reference images

## Troubleshooting

### Poor Quality Results
1. Use longer, more detailed prompts
2. Enable Skip Layer Guidance
3. Increase number of steps (30+)
4. Check reference image quality
5. Ensure proper mask creation

### Inconsistent Windows
1. Increase overlap frames
2. Use consistent prompting across windows
3. Add noise to overlapped frames
4. Reduce discard frames if losing too much content

### Memory Issues
1. Use VACE 1.3B instead of 13B
2. Reduce video length or resolution
3. Decrease window size
4. Enable quantization

### Blurry Results
1. Reduce overlap frames
2. Increase discard last frames
3. Use higher resolution reference images
4. Check control video quality

## Tips for Best Results

1. **Detailed Prompts**: Describe everything in the scene, especially elements not in reference images
2. **Quality Reference Images**: Use high-resolution, well-lit reference images
3. **Proper Masking**: Take time to create precise masks with Matanyone
4. **Iterative Approach**: Start with short videos, then extend successful results
5. **Background Preparation**: Remove complex backgrounds from object/person reference images
6. **Consistent Lighting**: Match lighting between reference images and intended scene