Community pipelines and components

Community pipelines are DiffusionPipeline classes that are different from the original paper implementation. They provide additional functionality or extend the original pipeline implementation.

Community pipelines are either stored on the Hub or the Diffusers’ GitHub repository. Hub pipelines are completely customizable (scheduler, models, pipeline code, etc.) while GitHub pipelines are limited to only the custom pipeline code. Further compare the two community pipeline types in the table below.

custom_pipeline

Load either community pipeline types by passing the custom_pipeline argument to from_pretrained().

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    custom_pipeline="pipeline_stable_diffusion_3_instruct_pix2pix",
    torch_dtype=torch.float16,
    device_map="cuda"
)

Add the custom_revision argument to from_pretrained() to load a community pipeline from a specific version (for example, v0.30.0 or main). By default, community pipelines are loaded from the latest stable version of Diffusers.

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    custom_pipeline="pipeline_stable_diffusion_3_instruct_pix2pix",
    custom_revision="main"
    torch_dtype=torch.float16,
    device_map="cuda"
)

There are a few ways to load a community pipeline.

• Pass a path to custom_pipeline to load a local community pipeline. The directory must contain a pipeline.py file containing the pipeline class.

  Copied

  import torch
  from diffusers import DiffusionPipeline
  
  pipeline = DiffusionPipeline.from_pretrained(
      "stabilityai/stable-diffusion-3-medium-diffusers",
      custom_pipeline="path/to/pipeline_directory",
      torch_dtype=torch.float16,
      device_map="cuda"
  )

• The custom_pipeline argument is also supported by from_pipe(), which is useful for reusing pipelines without using additional memory. It limits the memory usage to only the largest pipeline loaded.

  Copied

  import torch
  from diffusers import DiffusionPipeline
  
  pipeline_sd = DiffusionPipeline.from_pretrained("emilianJR/CyberRealistic_V3", torch_dtype=torch.float16, device_map="cuda")
  pipeline_lpw = DiffusionPipeline.from_pipe(
      pipeline_sd, custom_pipeline="lpw_stable_diffusion", device_map="cuda"
  )

  The from_pipe() method is especially useful for loading community pipelines because many of them don’t have pretrained weights. Community pipelines generally add a feature on top of an existing pipeline.

Community components

Community components let users build pipelines with custom transformers, UNets, VAEs, and schedulers not supported by Diffusers. These components require Python module implementations.

This section shows how users can use community components to build a community pipeline using showlab/show-1-base as an example.

1. Load the required components, the scheduler and image processor. The text encoder is generally imported from Transformers.

from transformers import T5Tokenizer, T5EncoderModel, CLIPImageProcessor
from diffusers import DPMSolverMultistepScheduler

pipeline_id = "showlab/show-1-base"
tokenizer = T5Tokenizer.from_pretrained(pipeline_id, subfolder="tokenizer")
text_encoder = T5EncoderModel.from_pretrained(pipeline_id, subfolder="text_encoder")
scheduler = DPMSolverMultistepScheduler.from_pretrained(pipe_id, subfolder="scheduler")
feature_extractor = CLIPImageProcessor.from_pretrained(pipe_id, subfolder="feature_extractor")

2. Load a custom UNet which is already implemented in showone_unet_3d_condition.py. The UNet3DConditionModel class name is renamed to the custom implementation, ShowOneUNet3DConditionModel, because UNet3DConditionModel already exists in Diffusers. Any components required for ShowOneUNet3DConditionModel class should be placed in showone_unet_3d_condition.py.

from showone_unet_3d_condition import ShowOneUNet3DConditionModel

unet = ShowOneUNet3DConditionModel.from_pretrained(pipeline_id, subfolder="unet")

3. Load the custom pipeline code (already implemented in pipeline_t2v_base_pixel.py). This script contains a custom TextToVideoIFPipeline class for generating videos from text. Like the custom UNet, any code required for TextToVideIFPipeline should be placed in pipeline_t2v_base_pixel.py.

Initialize TextToVideoIFPipeline with ShowOneUNet3DConditionModel.

import torch
from pipeline_t2v_base_pixel import TextToVideoIFPipeline

pipeline = TextToVideoIFPipeline(
    unet=unet,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    scheduler=scheduler,
    feature_extractor=feature_extractor,
    device_map="cuda",
    torch_dtype=torch.float16
)

4. Push the pipeline to the Hub to share with the community.

pipeline.push_to_hub("custom-t2v-pipeline")

After the pipeline is successfully pushed, make the following changes.

• Change the _class_name attribute in model_index.json to "pipeline_t2v_base_pixel" and "TextToVideoIFPipeline".
• Upload showone_unet_3d_condition.py to the unet subfolder.
• Upload pipeline_t2v_base_pixel.py to the pipeline repository.

To run inference, add the trust_remote_code argument while initializing the pipeline to handle all the “magic” behind the scenes.

import torch
from diffusers import DiffusionPipeline

pipeline = DiffusionPipeline.from_pretrained(
    "<change-username>/<change-id>", trust_remote_code=True, torch_dtype=torch.float16
)

Resources

• Take a look at Issue #841 for more context about why we’re adding community pipelines to help everyone easily share their work without being slowed down.
• Check out the stabilityai/japanese-stable-diffusion-xl repository for an additional example of a community pipeline that also uses the trust_remote_code feature.