metadata
base_model: black-forest-labs/FLUX.1-dev
library_name: diffusers
license: apache-2.0
tags:
- text-to-image
- diffusers-training
- diffusers
- lora
- FLUX.1-dev
- science
- materiomics
- bio-inspired
- materials science
- generative AI for science
datasets:
- lamm-mit/leaf-flux-images-and-captions
instance_prompt: <leaf microstructure>
widget: []
FLUX.1 [dev] Fine-tuned with Biological and Nature Images
FLUX.1 [dev] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions.
Install diffusers
pip install -U diffusers
Model description
These are LoRA adaption weights for the FLUX.1 [dev] model (black-forest-labs/FLUX.1-dev). The base model is, and you must first get access to it before loading this LoRA adapter.
This LoRA adapter has rank=64 and alpha=64, trained for 16,000 steps. Earlier checkpoints are available in this repository as well (you can load these via the adapter parameter, see example below).
Trigger keywords
The model was fine-tuned with a set of ~1,600 images of biological materials, structures, shapes and other images of nature, using the keyword <bioinspired>.
You should use <bioinspired> to trigger these features during image generation.
How to use
Defining some helper functions:
import os
from datetime import datetime
from PIL import Image
def generate_filename(base_name, extension=".png"):
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
return f"{base_name}_{timestamp}{extension}"
def save_image(image, directory, base_name="image_grid"):
filename = generate_filename(base_name)
file_path = os.path.join(directory, filename)
image.save(file_path)
print(f"Image saved as {file_path}")
def image_grid(imgs, rows, cols, save=True, save_dir='generated_images', base_name="image_grid",
save_individual_files=False):
if not os.path.exists(save_dir):
os.makedirs(save_dir)
assert len(imgs) == rows * cols
w, h = imgs[0].size
grid = Image.new('RGB', size=(cols * w, rows * h))
grid_w, grid_h = grid.size
for i, img in enumerate(imgs):
grid.paste(img, box=(i % cols * w, i // cols * h))
if save_individual_files:
save_image(img, save_dir, base_name=base_name+f'_{i}-of-{len(imgs)}_')
if save and save_dir:
save_image(grid, save_dir, base_name)
return grid
Text-to-image
Model loading:
from diffusers import FluxPipeline
import torch
repo_id = 'lamm-mit/bioinspired-L-FLUX.1-dev'
pipeline = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
torch_dtype=torch.bfloat16,
max_sequence_length=512,
)
#pipeline.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Comment out if you have enough GPU VRAM
adapter='bioinspired-flux.safetensors' #Step 16000, final step
#adapter='bioinspired-flux-step-11000.safetensors' #Step 11000, earlier checkpoint
pipeline.load_lora_weights(repo_id, weight_name=adapter) #You need to use the weight_name parameter since the repo includes multiple checkpoints
pipeline=pipeline.to('cuda')
Image generation - Example #1:
prompt="""Generate a futuristic, eco-friendly architectural concept utilizing a biomimetic composite material that integrates the structural efficiency of spider silk with the adaptive porosity of plant tissues. Utilize the following key features:
* Fibrous architecture inspired by spider silk, represented by sinuous lines and curved forms.
* Interconnected, spherical nodes reminiscent of plant cell walls, emphasizing growth and adaptation.
* Open cellular structures echoing the permeable nature of plant leaves, suggesting dynamic exchanges and self-regulation capabilities.
* Gradations of opacity and transparency inspired by the varying densities found in plant tissues, highlighting functional differentiation and multi-functionality.
"""
num_samples =2
num_rows = 2
n_steps=50
guidance_scale=3.5
all_images = []
for _ in range(num_rows):
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples,
guidance_scale=guidance_scale,).images
all_images.extend(image)
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, )
grid
Image generation - Example #2:
prompt = """A leaf during fall foliage, <bioinspired>."""
negative_prompt=""
num_samples =2
num_rows =2
n_steps=25
guidance_scale=5
all_images = []
for _ in range(num_rows):
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples,
guidance_scale=guidance_scale,).images
all_images.extend(image)
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, )
grid
Image generation - Example #3:
prompt = "A sign that says 'PDF to AUDIO' with organic shapes, <bioinspired>"
num_samples =1
num_rows =1
n_steps=25
guidance_scale=10
all_images = []
for _ in range(num_rows):
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples,
guidance_scale=guidance_scale,).images
all_images.extend(image)
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, )
grid
Image generation - Example #4:
prompt = "An architectural design in the style of <bioinspired>. The structure itself features key design elements as in <bioinspired>."
num_samples =1
num_rows =1
n_steps=50
guidance_scale=5.
all_images = []
for _ in range(num_rows):
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples,
guidance_scale=guidance_scale,).images
all_images.extend(image)
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, )
grid
Image generation - Example #5:
prompt = "<bioinspired>, a beautiful landscape."
num_samples =1
num_rows =1
n_steps=50
guidance_scale=5.0
all_images = []
for _ in range(num_rows):
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples,
guidance_scale=guidance_scale,).images
all_images.extend(image)
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, )
grid
Image generation - Example #6:
prompt = """A coffee mug in an unusual shape that resembles a <bioinspired> river during fall foliage."""
num_samples =1
num_rows =1
n_steps=50
guidance_scale=3.5
all_images = []
for _ in range(num_rows):
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples,
guidance_scale=guidance_scale,).images
all_images.extend(image)
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, )
grid
@article{BioinspiredFluxBuehler2024,
title={Fine-tuning image-generation models with biological patterns, shapes and topologies},
author={Markus J. Buehler},
journal={arXiv: XXXX.YYYYY},
year={2024},
}