Modular Pipeline: support for PixArtAlphaPipeline

[gasparyanartur]

Model/Pipeline/Scheduler description

I would like to implement a modular version of the PixArt pipeline family, as discussed in #13295.

The initial scope includes the PixArtAlphaPipeline under src/diffusers/pipelines/pixart_alpha/. Currently, the focus is on PixArt-α only. The implementation would be structured so that PixArtSigmaPipeline support can be added later in a follow-up.

Current plan:

Add a new modular PixArt package under src/diffusers/modular_pipelines/, following the structure of the existing built-in modular pipelines.

• encoders for prompt encoding
• before_denoise for input / latent preparation and timestep setup
• denoise for the main PixArt denoising loop
• decoders for VAE decoding and post-processing

I’m happy to implement this myself, and this would be my first contribution to Diffusers.

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Relevant references:

• Current Diffusers implementation:
  • src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py
  • src/diffusers/pipelines/pixart_alpha/pipeline_pixart_sigma.py
• PixArt docs / weights:
  • https://huggingface.co/docs/diffusers/api/pipelines/pixart
  • https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS

[cgloriacc]

Hi @sayakpaul @yiyixuxu — picking this up from #13295, where the modular PixArtAlphaPipeline port was already approved ("All good! Let's do this"). The original volunteer (@gasparyanartur) opened this tracking issue but it's been inactive for ~3 months with no PR, so I'd like to carry it forward if that's okay with everyone.

I've gone through the modular_diffusers docs and the stable_diffusion_3 / qwenimage modular implementations as references. Plan (text-to-image scope first, mirroring the sd3 block layout):

• encoders.py: PixArtAlphaTextEncoderStep — single T5, threading prompt_attention_mask; clean_caption preprocessing.
• inputs.py: text-input expansion + a step for the resolution-bin micro-conditions (resolution / aspect_ratio in added_cond_kwargs).
• before_denoise.py: SetTimestepsStep + PrepareLatentsStep.
• denoise.py: LoopDenoiser (CFG + PixArtTransformer2DModel forward with encoder_attention_mask + added_cond_kwargs) /LoopAfterDenoiser / DenoiseLoopWrapper.
• decoders.py: VAE decode + resolution un-binning.
• modular_blocks_pixart_alpha.py + modular_pipeline.py: assemble AUTO_BLOCKS (text_encoder → denoise → decode).

I'll keep PixArt-Sigma as a follow-up PR, and wait for a 👍 before opening the PR. Also — @sayakpaul, you mentioned some skills to bootstrap these implementations; could you point me to them?

[gasparyanartur]

Hi @sayakpaul @yiyixuxu — picking this up from #13295, where the modular PixArtAlphaPipeline port was already approved ("All good! Let's do this"). The original volunteer (@gasparyanartur) opened this tracking issue but it's been inactive for ~3 months with no PR, so I'd like to carry it forward if that's okay with everyone.

I've gone through the modular_diffusers docs and the stable_diffusion_3 / qwenimage modular implementations as references. Plan (text-to-image scope first, mirroring the sd3 block layout):

• encoders.py: PixArtAlphaTextEncoderStep — single T5, threading prompt_attention_mask; clean_caption preprocessing.
• inputs.py: text-input expansion + a step for the resolution-bin micro-conditions (resolution / aspect_ratio in added_cond_kwargs).
• before_denoise.py: SetTimestepsStep + PrepareLatentsStep.
• denoise.py: LoopDenoiser (CFG + PixArtTransformer2DModel forward with encoder_attention_mask + added_cond_kwargs) /LoopAfterDenoiser / DenoiseLoopWrapper.
• decoders.py: VAE decode + resolution un-binning.
• modular_blocks_pixart_alpha.py + modular_pipeline.py: assemble AUTO_BLOCKS (text_encoder → denoise → decode).

I'll keep PixArt-Sigma as a follow-up PR, and wait for a 👍 before opening the PR. Also — @sayakpaul, you mentioned some skills to bootstrap these implementations; could you point me to them?

Yes, that's okay with me, I have not been able to invest much time into this.

[cgloriacc]

Hi @sayakpaul @yiyixuxu — picking this up from #13295, where the modular PixArtAlphaPipeline port was already approved ("All good! Let's do this"). The original volunteer (@gasparyanartur) opened this tracking issue but it's been inactive for ~3 months with no PR, so I'd like to carry it forward if that's okay with everyone.
I've gone through the modular_diffusers docs and the stable_diffusion_3 / qwenimage modular implementations as references. Plan (text-to-image scope first, mirroring the sd3 block layout):

• encoders.py: PixArtAlphaTextEncoderStep — single T5, threading prompt_attention_mask; clean_caption preprocessing.
• inputs.py: text-input expansion + a step for the resolution-bin micro-conditions (resolution / aspect_ratio in added_cond_kwargs).
• before_denoise.py: SetTimestepsStep + PrepareLatentsStep.
• denoise.py: LoopDenoiser (CFG + PixArtTransformer2DModel forward with encoder_attention_mask + added_cond_kwargs) /LoopAfterDenoiser / DenoiseLoopWrapper.
• decoders.py: VAE decode + resolution un-binning.
• modular_blocks_pixart_alpha.py + modular_pipeline.py: assemble AUTO_BLOCKS (text_encoder → denoise → decode).

I'll keep PixArt-Sigma as a follow-up PR, and wait for a 👍 before opening the PR. Also — @sayakpaul, you mentioned some skills to bootstrap these implementations; could you point me to them?

Yes, that's okay with me, I have not been able to invest much time into this.

Thanks @gasparyanartur!
@sayakpaul @yiyixuxu — with the original author stepping aside, I'd like to pick up the modular PixArt-α port (approved in #13295). I'll start sketching it locally and open a draft PR for your feedback — happy to adjust the scope, or take a different model if you'd prefer.