--- base_model: black-forest-labs/FLUX.1-dev library_name: diffusers license: apache-2.0 tags: - text-to-image - diffusers-training - diffusers - lora - FLUX.1-dev - science - materiomics - bio-inspired - materials science - generative AI for science datasets: - lamm-mit/leaf-flux-images-and-captions instance_prompt: widget: [] --- # FLUX.1 [dev] Fine-tuned with Biological and Nature Images FLUX.1 [dev] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions. Install ```diffusers``` ```raw pip install -U diffusers ``` ## Model description These are LoRA adaption weights for the FLUX.1 [dev] model (```black-forest-labs/FLUX.1-dev```). The base model is, and you must first get access to it before loading this LoRA adapter. This LoRA adapter has rank=64 and alpha=64, trained for 16,000 steps. Earlier checkpoints are available in this repository as well (you can load these via the ```adapter``` parameter, see example below). ## Trigger keywords The model was fine-tuned with a set of ~1,600 images of biological materials, structures, shapes and other images of nature, using the keyword \. You should use \ to trigger these features during image generation. ## How to use Defining some helper functions: ```python import os from datetime import datetime from PIL import Image def generate_filename(base_name, extension=".png"): timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") return f"{base_name}_{timestamp}{extension}" def save_image(image, directory, base_name="image_grid"): filename = generate_filename(base_name) file_path = os.path.join(directory, filename) image.save(file_path) print(f"Image saved as {file_path}") def image_grid(imgs, rows, cols, save=True, save_dir='generated_images', base_name="image_grid", save_individual_files=False): if not os.path.exists(save_dir): os.makedirs(save_dir) assert len(imgs) == rows * cols w, h = imgs[0].size grid = Image.new('RGB', size=(cols * w, rows * h)) grid_w, grid_h = grid.size for i, img in enumerate(imgs): grid.paste(img, box=(i % cols * w, i // cols * h)) if save_individual_files: save_image(img, save_dir, base_name=base_name+f'_{i}-of-{len(imgs)}_') if save and save_dir: save_image(grid, save_dir, base_name) return grid ``` ### Text-to-image Model loading: ```python from diffusers import FluxPipeline import torch repo_id = 'lamm-mit/bioinspired-L-FLUX.1-dev' pipeline = FluxPipeline.from_pretrained( "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16, max_sequence_length=512, ) #pipeline.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Comment out if you have enough GPU VRAM adapter='bioinspired-flux.safetensors' #Step 16000, final step #adapter='bioinspired-flux-step-11000.safetensors' #Step 11000, earlier checkpoint pipeline.load_lora_weights(repo_id, weight_name=adapter) #You need to use the weight_name parameter since the repo includes multiple checkpoints pipeline=pipeline.to('cuda') ``` Image generation - Example #1: ```python prompt="""Generate a futuristic, eco-friendly architectural concept utilizing a biomimetic composite material that integrates the structural efficiency of spider silk with the adaptive porosity of plant tissues. Utilize the following key features: * Fibrous architecture inspired by spider silk, represented by sinuous lines and curved forms. * Interconnected, spherical nodes reminiscent of plant cell walls, emphasizing growth and adaptation. * Open cellular structures echoing the permeable nature of plant leaves, suggesting dynamic exchanges and self-regulation capabilities. * Gradations of opacity and transparency inspired by the varying densities found in plant tissues, highlighting functional differentiation and multi-functionality. """ num_samples =2 num_rows = 2 n_steps=50 guidance_scale=3.5 all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/qdnFLSWbzXOKeZdAUzNMH.png) Image generation - Example #2: ```python prompt = """A leaf during fall foliage, .""" negative_prompt="" num_samples =2 num_rows =2 n_steps=25 guidance_scale=5 all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/Vp47_0V7r3wLrDUcEBttV.png) Image generation - Example #3: ```python prompt = "A sign that says 'PDF to AUDIO' with organic shapes, " num_samples =1 num_rows =1 n_steps=25 guidance_scale=10 all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/kMTHiPszlXnZoBT-4klLY.png) Image generation - Example #4: ```python prompt = "An architectural design in the style of . The structure itself features key design elements as in ." num_samples =1 num_rows =1 n_steps=50 guidance_scale=5. all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/5t4cG5s7-Yf6bzBmNFqbQ.png) Image generation - Example #5: ```python prompt = ", a beautiful landscape." num_samples =1 num_rows =1 n_steps=50 guidance_scale=5.0 all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/ntM0EmRAD5E4ENman2k7e.png) Image generation - Example #6: ```python prompt = """A coffee mug in an unusual shape that resembles a river during fall foliage.""" num_samples =1 num_rows =1 n_steps=50 guidance_scale=3.5 all_images = [] for _ in range(num_rows): image = pipeline(prompt,num_inference_steps=n_steps,num_images_per_prompt=num_samples, guidance_scale=guidance_scale,).images all_images.extend(image) grid = image_grid(all_images, num_rows, num_samples, save_individual_files=True, ) grid ``` ![image/png](https://cdn-uploads.huggingface.co/production/uploads/623ce1c6b66fedf374859fe7/2fyOE9RG8TABnfzbgBI0p.png) ```bibtext @article{BioinspiredFluxBuehler2024, title={Fine-tuning image-generation models with biological patterns, shapes and topologies}, author={Markus J. Buehler}, journal={arXiv: XXXX.YYYYY}, year={2024}, }