|
--- |
|
base_model: black-forest-labs/FLUX.1-dev |
|
library_name: diffusers |
|
license: apache-2.0 |
|
tags: |
|
- text-to-image |
|
- diffusers-training |
|
- diffusers |
|
- lora |
|
- FLUX.1-dev |
|
- science |
|
- materiomics |
|
- bio-inspired |
|
- materials science |
|
- generative AI for science |
|
datasets: |
|
- lamm-mit/leaf-flux-images-and-captions |
|
instance_prompt: <leaf microstructure> |
|
widget: [] |
|
--- |
|
|
|
# FLUX.1 [dev] Fine-tuned with Biological and Nature Images |
|
|
|
FLUX.1 [dev] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions. |
|
|
|
Install ```diffusers``` |
|
|
|
```raw |
|
pip install -U diffusers |
|
``` |
|
|
|
## Model description |
|
|
|
These are LoRA adaption weights for the FLUX.1 [dev] model (```black-forest-labs/FLUX.1-dev```). The base model is, and you must first get access to it before loading this LoRA adapter. |
|
|
|
This LoRA adapter has rank=64 and alpha=64, trained for 16,000 steps. Earlier checkpoints are available in this repository as well (you can load these via the ```adapter``` parameter, see example below). |
|
|
|
## Trigger keywords |
|
|
|
The model was fine-tuned with a set of ~1,600 images of biological materials, structures, shapes and other images of nature, using the keyword \<bioinspired\>. |
|
|
|
You should use \<bioinspired\> to trigger these features during image generation. |
|
|
|
## How to use |
|
|
|
Defining some helper functions: |
|
|
|
```python |
|
import os |
|
from datetime import datetime |
|
from PIL import Image |
|
|
|
def generate_filename(base_name, extension=".png"): |
|
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") |
|
return f"{base_name}_{timestamp}{extension}" |
|
|
|
def save_image(image, directory, base_name="image_grid"): |
|
filename = generate_filename(base_name) |
|
file_path = os.path.join(directory, filename) |
|
image.save(file_path) |
|
print(f"Image saved as {file_path}") |
|
|
|
def image_grid(imgs, rows, cols, save=True, save_dir='generated_images', base_name="image_grid", |
|
save_individual_files=False): |
|
|
|
if not os.path.exists(save_dir): |
|
os.makedirs(save_dir) |
|
|
|
assert len(imgs) == rows * cols |
|
|
|
w, h = imgs[0].size |
|
grid = Image.new('RGB', size=(cols * w, rows * h)) |
|
grid_w, grid_h = grid.size |
|
|
|
for i, img in enumerate(imgs): |
|
grid.paste(img, box=(i % cols * w, i // cols * h)) |
|
if save_individual_files: |
|
save_image(img, save_dir, base_name=base_name+f'_{i}-of-{len(imgs)}_') |
|
|
|
if save and save_dir: |
|
save_image(grid, save_dir, base_name) |
|
|
|
return grid |
|
``` |
|
|
|
### Text-to-image |
|
|
|
Model loading: |
|
|
|
```python |
|
from diffusers import FluxPipeline |
|
import torch |
|
|
|
repo_id = 'lamm-mit/bioinspired-L-FLUX.1-dev' |
|
|
|
pipeline = FluxPipeline.from_pretrained( |
|
"black-forest-labs/FLUX.1-dev", |
|
torch_dtype=torch.bfloat16, |
|
max_sequence_length=512, |
|
) |
|
|
|
#pipeline.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Comment out if you have enough GPU VRAM |
|
|
|
adapter='bioinspired-flux.safetensors' #Step 16000, final step |
|
#adapter='bioinspired-flux-step-11000.safetensors' #Step 11000, earlier checkpoint |
|
|
|
pipeline.load_lora_weights(repo_id, weight_name=adapter) #You need to use the weight_name parameter since the repo includes multiple checkpoints |
|
|
|
pipeline=pipeline.to('cuda') |
|
``` |
|
|
|
Image generation - Example #1: |
|
|
|
```python |
|
prompt="""Generate a futuristic, eco-friendly architectural concept utilizing a biomimetic composite material that integrates the structural efficiency of spider silk with the adaptive porosity of plant tissues. Utilize the following key features: |
|
|
|
* Fibrous architecture inspired by spider silk, represented by sinuous lines and curved forms. |
|
* Interconnected, spherical nodes reminiscent of plant cell walls, emphasizing growth and adaptation. |
|
* Open cellular structures echoing the permeable nature of plant leaves, suggesting dynamic exchanges and self-regulation capabilities. |
|
* Gradations of opacity and transparency inspired by the varying densities found in plant tissues, highlighting functional differentiation and multi-functionality. |
|
""" |
|
|
|
num_samples =2 |
|
num_rows = 2 |
|
n_steps=50 |
|
guidance_scale=3.5 |
|
all_images = [] |
|
for _ in range(num_rows): |
|
|
|
|
|
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, |
|
guidance_scale=guidance_scale,).images |
|
|
|
all_images.extend(image) |
|
|
|
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) |
|
grid |
|
``` |
|
|
|
|
|
 |
|
|
|
Image generation - Example #2: |
|
|
|
```python |
|
prompt = """A leaf during fall foliage, <bioinspired>.""" |
|
negative_prompt="" |
|
num_samples =2 |
|
num_rows =2 |
|
n_steps=25 |
|
guidance_scale=5 |
|
all_images = [] |
|
for _ in range(num_rows): |
|
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, |
|
guidance_scale=guidance_scale,).images |
|
|
|
all_images.extend(image) |
|
|
|
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) |
|
grid |
|
``` |
|
|
|
 |
|
|
|
Image generation - Example #3: |
|
|
|
```python |
|
prompt = "A sign that says 'PDF to AUDIO' with organic shapes, <bioinspired>" |
|
num_samples =1 |
|
num_rows =1 |
|
n_steps=25 |
|
guidance_scale=10 |
|
all_images = [] |
|
for _ in range(num_rows): |
|
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, |
|
guidance_scale=guidance_scale,).images |
|
|
|
all_images.extend(image) |
|
|
|
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) |
|
grid |
|
``` |
|
|
|
 |
|
|
|
Image generation - Example #4: |
|
|
|
```python |
|
prompt = "An architectural design in the style of <bioinspired>. The structure itself features key design elements as in <bioinspired>." |
|
num_samples =1 |
|
num_rows =1 |
|
n_steps=50 |
|
guidance_scale=5. |
|
all_images = [] |
|
for _ in range(num_rows): |
|
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, |
|
guidance_scale=guidance_scale,).images |
|
|
|
all_images.extend(image) |
|
|
|
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) |
|
grid |
|
``` |
|
|
|
 |
|
|
|
|
|
Image generation - Example #5: |
|
|
|
```python |
|
prompt = "<bioinspired>, a beautiful landscape." |
|
num_samples =1 |
|
num_rows =1 |
|
n_steps=50 |
|
guidance_scale=5.0 |
|
all_images = [] |
|
for _ in range(num_rows): |
|
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, |
|
guidance_scale=guidance_scale,).images |
|
|
|
all_images.extend(image) |
|
|
|
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) |
|
grid |
|
``` |
|
|
|
 |
|
|
|
|
|
Image generation - Example #6: |
|
|
|
```python |
|
prompt = """A coffee mug in an unusual shape that resembles a <bioinspired> river during fall foliage.""" |
|
num_samples =1 |
|
num_rows =1 |
|
n_steps=50 |
|
guidance_scale=3.5 |
|
all_images = [] |
|
for _ in range(num_rows): |
|
image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, |
|
guidance_scale=guidance_scale,).images |
|
|
|
all_images.extend(image) |
|
|
|
grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) |
|
grid |
|
``` |
|
|
|
 |
|
|
|
```bibtext |
|
@article{BioinspiredFluxBuehler2024, |
|
title={Fine-tuning image-generation models with biological patterns, shapes and topologies}, |
|
author={Markus J. Buehler}, |
|
journal={arXiv: XXXX.YYYYY}, |
|
year={2024}, |
|
} |
