Creating longer videos can be a challenging task for diffusion-based models, but with Pyramid Flow, it has become much more manageable. This innovative text-to-video model leverages technologies like Stable Diffusion 3 Medium, CogVideoX, Flux 1.0, and more to enable users to generate high-quality videos. In this blog post, we’ll guide you through the process of generating 10-second videos using Pyramid Flow on Comfy UI.
In this blog post, we will guide you through the steps to generate 10-second videos using Pyramid Flow on Comfy UI, helping you unlock the potential of this powerful tool.
What is Pyramid Flow?
Pyramid Flow is an open-source video generation model developed in collaboration with Kuaishou, Peking University, and Beijing University of Posts and Telecommunications. It can produce videos up to 10 seconds long at resolutions of 1280x768 pixels and a frame rate of 24 FPS. This capability significantly enhances the potential for creative video production, making it especially accessible for users with a 12 GB VRAM equipped GPU.
Installation Steps
To start using Pyramid Flow, you need to install the appropriate plugin. Here’s how to do it:
Download the Plugin:
Visit the Pyramid Flow GitHub repository to download the plugin. Please note that it hasn’t been added to the Comfy UI manager yet, so you’ll need to download it manually.
Set Up the Model:
The first time you run the workflow, the model will automatically download. The total size is approximately 27 GB, so ensure you have sufficient storage.
2.1 Install the Nodes:
Open ComfyUI on MimicPC , Move to "Storage>custom_nodes" folder, upload the nodes.
2.2 Install the model:
Move to "Storage>models> pyramidflow" folder, upload the model with Hugginface link.
Use Existing Workflows:
If you prefer not to download the model yourself, MimicPC has already uploaded workflow images for text-to-video and image-to-video generation. You can access these workflows directly via the MimicPC Discord link.
Text2video
The Pyramid Flow workflow offers versatile options for video generation, allowing you to tailor your content creation to your specific needs. With Pyramid Flow, you can choose to generate videos using a single prompt, which is ideal for straightforward video creation based on a simple idea or theme. Alternatively, you can utilize multiple prompts to generate more complex and dynamic videos, combining various concepts and visual elements to enhance creativity. Additionally, Pyramid Flow supports image-to-video generation, enabling users to create videos that bring still images to life, offering a seamless transition between static and dynamic content.
Image2video
Multi prompt2video
a campfire burning with flames and embers,gradually increasing in size and intensity before dying down towards the end,hyper quality,Ultra HD,8K
A massive explosion on the surface of the earth, hyper qulity,Ultra HD,8K
cartoon style, worst quality, low quality, blurry, absolute black, absolute white, low res, extra limbs, extra digits, misplaced objects, mutated anatomy, monochrome, horror
Generating Image-to-Video
Model Loading Node
- Variant Selection: You can choose between two variants: 384p and 768p. The 768p variant uses the default resolution of 1280x768. If you select 384p, the resolution should be adjusted to 640x384.
- Precision (dtype): The ‘dtype’ parameter allows you to adjust the model's precision. Using fp8 can save memory, which is helpful if you face memory constraints.
- Temp Settings: Adjusting the ‘temp’ setting is crucial for video length. Set temp=16 for a 5-second video. If you want to generate a 10-second video, change it to temp=31.
Recommended Settings for Text to Video
For optimal video quality, use the following settings:
- Number of Inference Steps:
num inference steps = 20, 20, 20 - Video Inference Steps:
video num inference steps = 10, 10, 10 - Height and Width:
height = 768,width = 1280 - Guidance Scale:
guidance scale = 7.0 - Video Guidance Scale:
video guidance scale = 4.0 - Temp Setting:
temp = 16
Generating Image-to-Video
Generating videos from images follows a similar process, with a few parameter tweaks.
Recommended Settings for Image to Video
- Number of Inference Steps:
num inference steps = 10, 10, 10 - Temp: Set to
temp = 16 - Video Guidance Scale:
video guidance scale = 4.0
While the model performs exceptionally well with landscapes, it may struggle with human figures, so keep this in mind when creating content.
Conclusion
Pyramid Flow offers an exciting way to generate high-quality videos efficiently. While the landscapes it produces are impressive, the model is continuously improving, particularly in rendering human figures. We encourage interested users to explore Pyramid Flow and experiment with video generation techniques in Comfy UI. For those looking for an accessible method, using pre-configured workflows from the MimicPC Discord can save time and effort. Happy video creation!