Mirrors/AnimateDiff
1
0
Fork 0
mirror of https://github.com/guoyww/AnimateDiff.git synced 2025-12-16 16:38:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

reorg

Browse Source
This commit is contained in:
Yuwei Guo
2023-12-16 01:23:33 +08:00
parent 212ebb6a80
commit 57e7d14ede
1 changed files with 3 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@@ -66,11 +66,12 @@ Manually download the AnimateDiff modules. The download links can be found in ea
In this version, we did the image model finetuning through **Domain Adapter LoRA** for more flexiblity at inference time.
Additionally, we implement two (RGB image/scribble) [SparseCtrl](https://arxiv.org/abs/2311.16933) Encoders, which can take abitary number of condition maps to control the generation process.
- **Explanation:** Domain Adapter is a LoRA module trained on static frames of the training video dataset. This process is done before training the motion module, and helps the motion module focus on motion modeling, as shown in the figure below. At inference, By adjusting the LoRA scale of the Domain Adapter, some visual attributes of the training video, e.g., the watermarks, can be removed. To utilize the SparseCtrl encoder, it's necessary to use a full Domain Adapter in the pipeline.
<div style="text-align:center"><img src="__assets__/figs/adapter_explain.png" style="width:60%"></div>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<img src="__assets__/figs/adapter_explain.png" style="width:60%">
Additionally, we implement two (RGB image/scribble) [SparseCtrl](https://arxiv.org/abs/2311.16933) Encoders, which can take abitary number of condition maps to control the generation process.
Technical details of SparseCtrl can be found in this research paper: