how to use Hyper-SD15-8steps-CFG-LoRA

#27
by michaelj - opened

When I use this model Hyper-SD15-8steps CFG LoRA, why is there no change in quality and speed, even if the image quality becomes worse and the speed does not increase? Is there a problem with my usage method
import torch
from PIL import Image
from diffusers import ControlNetModel,DDIMScheduler, StableDiffusionControlNetImg2ImgPipeline
from diffusers.utils import load_image
from huggingface_hub import hf_hub_download
repo_name = "ByteDance/Hyper-SD"

Take 2-steps lora as an example

ckpt_name = "Hyper-SD15-8steps-CFG-lora.safetensors"

def resize_for_condition_image(input_image: Image, resolution: int):
input_image = input_image.convert("RGB")
W, H = input_image.size
k = float(resolution) / max(H, W)
H *= k
W *= k
H = int(round(H / 64.0)) * 64
W = int(round(W / 64.0)) * 64
img = input_image.resize((W, H), resample=Image.LANCZOS)
return img

controlnet = ControlNetModel.from_pretrained('lllyasviel/control_v11f1e_sd15_tile',
torch_dtype=torch.float16)
tiles_pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained("philz1337/epicrealism",
controlnet=controlnet,
torch_dtype=torch.float16).to('cuda')
scheduler = DDIMScheduler.from_config(pipe.scheduler.config)

tiles_pipe.scheduler = scheduler
tiles_pipe.load_lora_weights(hf_hub_download(repo_name, ckpt_name))
tiles_pipe.fuse_lora()
tiles_pipe.enable_xformers_memory_efficient_attention()
tiles_pipe.enable_vae_slicing()
tiles_pipe.enable_vae_tiling()
import time
start=time.time()
source_image = load_image('https://img0.baidu.com/it/u=2546412824,3393178378&fm=253&fmt=auto&app=120&f=JPEG?w=900&h=600')

condition_image = resize_for_condition_image(source_image, 1024)

torch.cuda.empty_cache()

pipe = tiles_pipe(prompt="blurry, illustration, drawing, painting, over saturated, overexposed,",
negative_prompt="blurry, illustration, drawing, painting, over saturated, overexposed, tattoos, chocker, wig",
image=condition_image,
control_image=condition_image,
width=condition_image.size[0],
height=condition_image.size[1],
strength=1,
generator=torch.manual_seed(0),
num_inference_steps=30,
).images[0]

pipe.save('a.png')
end=time.time()
print("time",end-start)

use Hyper-SD15-8steps-CFG-LoRA image
46c1dba8-f0b0-4d1a-9fe5-c87d95851cb0.png
but Not used is better

a.png

ByteDance org

@michaelj , you need to set num_inference_steps=8 and adjust the lora weight and guidance_scale to get better results and acceleration.

renyuxi changed discussion status to closed

Sign up or log in to comment