Any chance of docs regarding on best practices for fine-tuning for SDXL-Lightning?
It would be great to get some guidance on how to best fine-tune for SDXL-Lightning. It would help the community really explore what SDXL-Lightning could do, especially for image2video and video2video workflows.
Thanks for the awesome release.
Yes please
Releated: https://huggingface.co/ByteDance/SDXL-Lightning/discussions/11
As for video, that's future research topic. The method used in SDXL-Lightning called progressive adversarial distillation can be used for video models. But that is beyond simple finetuning.
Yeah I was thinking something along the lines of AnimateLightning, akin to AnimateLCM for vid2vid. Or optimizing AnimateDiff to use SDXL-Lightning.
I had a look at the link you provided: https://huggingface.co/ByteDance/SDXL-Lightning/discussions/11
- While your most of your suggestions are pretty doable for most people in the fine-tuning community. I think what everyone wants to know how to do(code) is: The most advanced is to merge SDXL-Lightning LoRA, then use adversarial objective, like the SDXL-Lightning paper does.
- It would be cool to incorporate that into the diffusers library for training scripts, especially the "use adverserial objective".