About This Repository
This repository has been established to house the controlnet models that have been corrected by the Diffus team. These models are now compatible with the sd-webui-controlnet extension.
About Diffus
Diffus provides an online Stable Diffusion webui service. The core UI is a heavily modified version of the original AUTOMATIC1111/stable-diffusion-webui. All modifications have been open-sourced in accordance with the original GPL license.
About the Fixes
In the process of integrating controlnet with our platform, we identified issues in some widely used models that prevented them from working seamlessly with the sd-webui-controlnet extension. We diagnosed these issues, made necessary modifications to the models, and now wish to share our solutions with the broader community. That's the purpose of this repository.
1. Issues with t2i-adapter_diffusers_xl_ Series Models
The original models can be found at lllyasviel/sd_control_collection.
A common error encountered with these models is:
RuntimeError: Error(s) in loading state_dict for Adapter:
size mismatch for body.2.in_conv.weight: copying a param with shape torch.Size([1280, 640, 1, 1]) from checkpoint, the shape in current model is torch.Size([640, 320, 1, 1]).
size mismatch for body.2.in_conv.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([640]).
This error arises due to a mismatch between the keys in the trained models and those defined for the controlnet xl t2i-adapter. This mismatch renders the following models incompatible with sd-webui-controlnet (v1.1.4):
- t2i-adapter_diffusers_xl_canny.safetensors
- t2i-adapter_diffusers_xl_depth_midas.safetensors
- t2i-adapter_diffusers_xl_depth_zoe.safetensors
- t2i-adapter_diffusers_xl_lineart.safetensors
- t2i-adapter_diffusers_xl_openpose.safetensors
- t2i-adapter_diffusers_xl_sketch.safetensors
We have corrected the keys in these models and repackaged them as safetensors. You can download these updated models for use in your local installation of the Automatic1111 stable diffusion webui. They should be placed in the following location:
stable-diffusion-webui/extensions/sd-webui-controlnet/models
Alternatively, you can use these models directly via our online service, eliminating the need to maintain your own instance.