Optimizing Stable Diffusion for macOS: A Comprehensive Guide

Introduction

Stable Diffusion has transformed AI-generated imagery, but running it efficiently on macOS can be tricky. Many popular interfaces, like AUTOMATIC1111’s web UI, don’t fully leverage Apple Silicon’s Metal Performance Shaders (MPS), resulting in subpar performance. This guide will show you how to use the Hugging Face Diffusers library to optimize Stable Diffusion on macOS, focusing on the Stable Diffusion XL model.

Why Choose Diffusers?

The Hugging Face Diffusers library offers several advantages for macOS users:

  1. Efficient MPS utilization on Apple Silicon Macs
  2. Greater control over the generation process
  3. Flexibility to experiment with various models and schedulers

Setting Up Your Environment

Begin by creating a Python virtual environment and installing the necessary libraries:

python3 -m venv venv
source venv/bin/activate
pip install torch transformers diffusers

Implementing Stable Diffusion XL

Here’s a Python script that demonstrates how to use Stable Diffusion XL with MPS on macOS:

import torch
from diffusers import StableDiffusionXLPipeline, EulerDiscreteScheduler

# Configuration
WIDTH, HEIGHT = 512, 896
NUM_INFERENCE_STEPS = 30
NUM_IMAGES_PER_PROMPT = 4
MODEL_PATH = "./sd_xl_base_1.0.safetensors" # Path to your .safetensors checkpoint file

# Initialize pipeline
def create_pipeline(model_path, scheduler=None):
    return StableDiffusionXLPipeline.from_single_file(
        model_path,
        torch_dtype=torch.float32,
        variant="fp32",
        use_safetensors=True,
        scheduler=scheduler
    )

# Set up pipeline
pipe = create_pipeline(MODEL_PATH)
scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
pipe = create_pipeline(MODEL_PATH, scheduler)

# Optimize for macOS
pipe.to("mps")
pipe.enable_attention_slicing()

# Generate images
def generate_images(prompt, negative_prompt=""):
    return pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        height=HEIGHT,
        width=WIDTH,
        num_inference_steps=NUM_INFERENCE_STEPS,
        num_images_per_prompt=NUM_IMAGES_PER_PROMPT,
        generator=torch.Generator(device="mps")
    ).images

# Example usage
prompt = "a photo of an astronaut riding a horse on mars"
negative_prompt = "blurry, dark photo, blue"
images = generate_images(prompt, negative_prompt)

# Save images
for i, image in enumerate(images):
    image.save(f"./output-{i:02d}.png")

Key Components Explained

  1. Pipeline Initialization: We use StableDiffusionXLPipeline to load a custom model (.safetensors file). We set torch_dtype=torch.float32 because MPS doesn’t support fp16.

  2. Scheduler: We employ the EulerDiscreteScheduler, known for producing quality results with Stable Diffusion.

  3. MPS Utilization: We move the pipeline to the MPS device with pipe.to("mps").

  4. Prompt Engineering: We define both positive and negative prompts to guide image generation.

  5. Image Generation: We call the pipeline with our defined parameters, including dimensions, inference steps, and images per prompt.

Using StableDiffusionXLPipeline with .safetensors Files

When working with Stable Diffusion XL models in .safetensors format, it’s crucial to use the StableDiffusionXLPipeline instead of the regular StableDiffusionPipeline.

To use a .safetensors file with StableDiffusionXLPipeline:

  1. Specify the file path in MODEL_PATH
  2. Use the from_single_file() method to load the model
  3. Set use_safetensors=True when initializing the pipeline

Here’s the relevant part of the code:

MODEL_PATH = "./sd_xl_base_1.0.safetensors"

def create_pipeline(model_path, scheduler=None):
    return StableDiffusionXLPipeline.from_single_file(
        model_path,
        torch_dtype=torch.float32,
        variant="fp32",
        use_safetensors=True,
        scheduler=scheduler
    )

This approach ensures that you’re using the correct pipeline for SDXL models and properly loading the .safetensors file.

Sourcing .safetensors Files

.safetensors files are widely available from various sources, allowing you to experiment with different models and fine-tuned versions of Stable Diffusion. Here are some popular places to find .safetensors files:

  1. Civitai.com: This is a community-driven platform where artists and developers share their custom models, including many in .safetensors format. It’s an excellent resource for finding specialized or themed models.

  2. Hugging Face Model Hub: Many official and community-contributed models are available here, often in .safetensors format.

When downloading .safetensors files, keep in mind:

  • Always respect the licenses and terms of use associated with each model.
  • Be cautious when downloading from unofficial sources and scan files for potential security risks.
  • Check the model card or documentation for specific instructions on how to use the model, as some may require special prompts or settings.

Customization and Experimentation

Feel free to experiment with:

  • Different models by changing the MODEL_PATH
  • Other schedulers from the Diffusers library
  • Adjusting WIDTH, HEIGHT, NUM_INFERENCE_STEPS, and NUM_IMAGES_PER_PROMPT
  • Crafting various prompts and negative prompts

Conclusion

By leveraging the Hugging Face Diffusers library and using the appropriate StableDiffusionXLPipeline, we can run Stable Diffusion XL efficiently on macOS, taking full advantage of Metal Performance Shaders. This approach provides fine-grained control over the generation process and allows for easy experimentation with different models, schedulers, and parameters.

Remember to respect the licenses of the models you use. Happy exploring in the world of AI-generated images on your Mac!