update readme
Browse files
README.md
ADDED
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# InstructPix2Pix for AnimateDiff
|
2 |
+
|
3 |
+
This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model.
|
4 |
+
|
5 |
+
## Inference
|
6 |
+
|
7 |
+
You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is [here](https://github.com/viddle-app/viddle-pix2pix-animatediff)
|
8 |
+
|
9 |
+
Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.
|
10 |
+
|
11 |
+
### Init Image
|
12 |
+
|
13 |
+
![Init Image](/images/mona-lisa.jpg)
|
14 |
+
|
15 |
+
_Make her smile_
|
16 |
+
|
17 |
+
![Smiling](/images/mona-lisa-smiles.mp4)
|
18 |
+
|
19 |
+
_Make her on fire_
|
20 |
+
|
21 |
+
![On Fire](/images/mona-lisa-on-fire.mp4)
|
22 |
+
|
23 |
+
## Training
|
24 |
+
The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1.
|
25 |
+
|
26 |
+
## LICENSE
|
27 |
+
Creative OpenRAIL License
|