feat: Add Motif-Video model and pipelines#13551
feat: Add Motif-Video model and pipelines#13551waitingcheung wants to merge 1 commit intohuggingface:mainfrom
Conversation
|
Quick ping for visibility. This PR adds Motif-Video (T2V/I2V + new transformer and pipelines). Would appreciate your feedback, especially on dependency/version constraints:
This is currently blocking some diffusers-side integration, so your input would help. A working branch for this integration is available here. |
…dance support Add complete Motif Video implementation to diffusers: New Models: - Add MotifVideoTransformer3DModel with T5Gemma2Encoder for multimodal conditioning - Supports text-to-video and image-to-video generation with vision tower integration New Pipelines: - Add MotifVideoPipeline for text-to-video generation - Default resolution: 736x1280, 121 frames, 25 fps - Supports classifier-free guidance and AdaptiveProjectedGuidance - Add MotifVideoImage2VideoPipeline for image-to-video generation - First frame conditioning with vision encoder - Same defaults as T2V pipeline Enhanced Guidance: - Update AdaptiveProjectedGuidance with normalization_dims parameter - Support "spatial" normalization for 5D tensors (per-frame spatial normalization) - Support custom dimension lists for flexible normalization - Update AdaptiveProjectedMixGuidance with same parameter Documentation & Tests: - Add comprehensive API documentation for transformer and pipelines - Add test suites for both T2V and I2V pipelines - Register all new components in __init__ files - Add dummy objects for torch and transformers backends Total: 18 files changed, 3416 insertions(+), 2 deletions(-)
cd20ffc to
81cce23
Compare
I think we can guard the transformers import in the pipeline with something like
|
We have something like this at the top of the pipeline code to guide the users to upgrade the # Check transformers version before importing T5Gemma2Encoder
if not is_transformers_version(">=", "5.1.0"):
import transformers
raise ImportError(
f"MotifVideoPipeline requires transformers>=5.1.0. "
f"Found: {transformers.__version__}. "
"Please upgrade transformers: pip install transformers --upgrade"
) |
|
Then it will cut it. |
What does this PR do?
This PR adds support for Motif-Video - a text-to-video (T2V) and image-to-video (I2V) diffusion model from Motif Technologies. The implementation includes the transformer architecture, both pipeline variants, guiding configurations, and comprehensive documentation.
Changes
New Files
src/diffusers/models/transformers/transformer_motif_video.py- MotifVideoTransformer3DModelsrc/diffusers/pipelines/motif_video/pipeline_motif_video.py- Text-to-Videosrc/diffusers/pipelines/motif_video/pipeline_motif_video_image2video.py- Image-to-Videosrc/diffusers/pipelines/motif_video/pipeline_output.pytests/pipelines/motif_video/test_motif_video.pytests/pipelines/motif_video/test_motif_video_image2video.pydocs/source/en/api/models/motif_video_transformer_3d.mddocs/source/en/api/pipelines/motif_video.mdKey Features
Version Requirements
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.