mirror of
https://github.com/guoyww/AnimateDiff.git
synced 2026-04-03 01:36:20 +02:00
111 lines
4.1 KiB
Markdown
111 lines
4.1 KiB
Markdown
# AnimateDiff: training and inference setup
|
|
## Setups for Inference
|
|
|
|
### Prepare Environment
|
|
|
|
***We updated our inference code with xformers and a sequential decoding trick. Now AnimateDiff takes only ~12GB VRAM to inference, and run on a single RTX3090 !!***
|
|
|
|
```
|
|
git clone https://github.com/guoyww/AnimateDiff.git
|
|
cd AnimateDiff
|
|
|
|
conda env create -f environment.yaml
|
|
conda activate animatediff
|
|
```
|
|
|
|
### Download Base T2I & Motion Module Checkpoints
|
|
We provide two versions of our Motion Module, which are trained on stable-diffusion-v1-4 and finetuned on v1-5 seperately.
|
|
It's recommanded to try both of them for best results.
|
|
```
|
|
git lfs install
|
|
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/
|
|
|
|
bash download_bashscripts/0-MotionModule.sh
|
|
```
|
|
You may also directly download the motion module checkpoints from [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI?usp=sharing) / [HuggingFace](https://huggingface.co/guoyww/animatediff) / [CivitAI](https://civitai.com/models/108836/animatediff-motion-modules), then put them in `models/Motion_Module/` folder.
|
|
|
|
### Prepare Personalize T2I
|
|
Here we provide inference configs for 6 demo T2I on CivitAI.
|
|
You may run the following bash scripts to download these checkpoints.
|
|
```
|
|
bash download_bashscripts/1-ToonYou.sh
|
|
bash download_bashscripts/2-Lyriel.sh
|
|
bash download_bashscripts/3-RcnzCartoon.sh
|
|
bash download_bashscripts/4-MajicMix.sh
|
|
bash download_bashscripts/5-RealisticVision.sh
|
|
bash download_bashscripts/6-Tusun.sh
|
|
bash download_bashscripts/7-FilmVelvia.sh
|
|
bash download_bashscripts/8-GhibliBackground.sh
|
|
```
|
|
|
|
### Inference
|
|
After downloading the above peronalized T2I checkpoints, run the following commands to generate animations. The results will automatically be saved to `samples/` folder.
|
|
```
|
|
python -m scripts.animate --config configs/prompts/1-ToonYou.yaml
|
|
python -m scripts.animate --config configs/prompts/2-Lyriel.yaml
|
|
python -m scripts.animate --config configs/prompts/3-RcnzCartoon.yaml
|
|
python -m scripts.animate --config configs/prompts/4-MajicMix.yaml
|
|
python -m scripts.animate --config configs/prompts/5-RealisticVision.yaml
|
|
python -m scripts.animate --config configs/prompts/6-Tusun.yaml
|
|
python -m scripts.animate --config configs/prompts/7-FilmVelvia.yaml
|
|
python -m scripts.animate --config configs/prompts/8-GhibliBackground.yaml
|
|
```
|
|
|
|
To generate animations with a new DreamBooth/LoRA model, you may create a new config `.yaml` file in the following format:
|
|
```
|
|
- inference_config: "[path to motion module config file]"
|
|
|
|
motion_module:
|
|
- "models/Motion_Module/mm_sd_v14.ckpt"
|
|
- "models/Motion_Module/mm_sd_v15.ckpt"
|
|
|
|
motion_module_lora_configs:
|
|
- path: "[path to MotionLoRA model]"
|
|
alpha: 1.0
|
|
- ...
|
|
|
|
dreambooth_path: "[path to your DreamBooth model .safetensors file]"
|
|
lora_model_path: "[path to your LoRA model .safetensors file, leave it empty string if not needed]"
|
|
|
|
steps: 25
|
|
guidance_scale: 7.5
|
|
|
|
prompt:
|
|
- "[positive prompt]"
|
|
|
|
n_prompt:
|
|
- "[negative prompt]"
|
|
```
|
|
Then run the following commands:
|
|
```
|
|
python -m scripts.animate --config [path to the config file]
|
|
```
|
|
|
|
|
|
## Steps for Training
|
|
|
|
### Dataset
|
|
Before training, download the videos files and the `.csv` annotations of [WebVid10M](https://maxbain.com/webvid-dataset/) to the local mechine.
|
|
Note that our examplar training script requires all the videos to be saved in a single folder. You may change this by modifying `animatediff/data/dataset.py`.
|
|
|
|
### Configuration
|
|
After dataset preparations, update the below data paths in the config `.yaml` files in `configs/training/` folder:
|
|
```
|
|
train_data:
|
|
csv_path: [Replace with .csv Annotation File Path]
|
|
video_folder: [Replace with Video Folder Path]
|
|
sample_size: 256
|
|
```
|
|
Other training parameters (lr, epochs, validation settings, etc.) are also included in the config files.
|
|
|
|
### Training
|
|
To finetune the unet's image layers
|
|
```
|
|
torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/training/v1/image_finetune.yaml
|
|
```
|
|
|
|
To train motion modules
|
|
```
|
|
torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/training/v1/training.yaml
|
|
```
|