Introducing Modular Diffusers - Composable Building Blocks for Diffusion Pipelines

Modular Diffusers introduces a new way to build diffusion pipelines by composing reusable blocks. Instead of writing entire pipelines from scratch, you can mix and match blocks to create workflows tailored to your needs! This complements the existing DiffusionPipeline class with a more flexible, composable alternative.

In this post, we'll walk through how Modular Diffusers works — from the familiar API to run a modular pipeline, to building fully custom blocks and composing them into your own workflow. We'll also show how it integrates with Mellon, a node-based visual workflow interface that you can use to wire Modular Diffusers blocks together.

Here is a simple example of how to run inference with FLUX.2 Klein 4B using pre-built blocks:

ModularPipeline.from_pretrained works with any existing Diffusers repo out of the box, but Modular Diffusers also introduces a new kind of repo: the Modular Repository.

A modular repository is able to reference components from their original model repos. For example, diffusers/flux2-bnb-4bit-modular contains a quantized transformer and loads the remaining components from the original repo.

Each block is self-contained with its own inputs and outputs. You can run any block independently as its own pipeline, or add, remove, and swap blocks freely — they dynamically recompose to work with whatever blocks remain. Use .init_pipeline() to convert blocks into a runnable pipeline, and .load_components() to load the model weights.

The community has already started building complete pipelines with Modular Diffusers and publishing them on the Hub, with model weights and ready-to-run code.

Modular Diffusers really shines when creating your own blocks. A custom block is a Python class that defines its components, inputs, outputs, and computation logic — and once defined, you can plug it into any workflow.

Let's use this block with Qwen's ControlNet workflow. Extract the ControlNet workflow and insert the depth block at the beginning:

For example, diffusers/FLUX.2-klein-4B-modular contains a pipeline definition, component references, and a mellon_pipeline_config.json — all in one repo. Load it in Python with ModularPipeline.from_pretrained("diffusers/FLUX.2-klein-4B-modular") or in Mellon to create either a single-node or multi-node workflow.

You get the same results as with a standard DiffusionPipeline , but the pipeline is very different under the hood: it's composed of flexible blocks — text encoding, image encoding, denoising, and decoding — that you can inspect directly:

For more on block types, composition patterns, lazy loading, and memory management with ComponentsManager , check out the Modular Diffusers documentation .

Here's an example block that extracts depth maps from images using Depth Anything V2 .