可生成高清视频的Stable Diffusion。

Nathan Raw 79cb96699e Merge pull request #154 from philgzl/recursive-upsampling 5 days ago
.github 23e04894f7 CD: allow from draft 4 months ago
examples 564a33a301 Update make_music_video.py 3 months ago
stable_diffusion_videos 9de2a48f3c Update stable_diffusion_videos/upsampling.py 5 days ago
tests cff9b90295 :art: cleanup and fix tests 3 months ago
.gitignore e5d9c2f8ab :see_no_evil: add music files to gitignore 1 month ago
LICENSE aedfacb243 Initial commit 4 months ago
README.md 23a726a19b Update README.md 1 week ago
flax_stable_diffusion_videos.ipynb f20a0b3b33 Created using Colaboratory 1 week ago
packages.txt a1244dc452 :tada: init 4 months ago
requirements.txt c2f0423c97 Update requirements.txt 3 weeks ago
setup.py 1685670145 :pushpin: move realesrgan to required deps 1 month ago
stable_diffusion_videos.ipynb 8c77e29f8a Created using Colaboratory 1 month ago

README.md

stable-diffusion-videos

Try it yourself in Colab: Open In Colab

TPU version (~x6 faster than standard colab GPUs): Open In Colab

Example - morphing between "blueberry spaghetti" and "strawberry spaghetti"

https://user-images.githubusercontent.com/32437151/188721341-6f28abf9-699b-46b0-a72e-fa2a624ba0bb.mp4

How it Works

The Notebook/App

The in-browser Colab demo allows you to generate videos by interpolating the latent space of Stable Diffusion.

You can either dream up different versions of the same prompt, or morph between different text prompts (with seeds set for each for reproducibility).

The app is built with Gradio, which allows you to interact with the model in a web app. Here's how I suggest you use it:

  1. Use the "Images" tab to generate images you like.

    • Find two images you want to morph between
    • These images should use the same settings (guidance scale, scheduler, height, width)
    • Keep track of the seeds/settings you used so you can reproduce them
  2. Generate videos using the "Videos" tab

    • Using the images you found from the step above, provide the prompts/seeds you recorded
    • Set the num_interpolation_steps - for testing you can use a small number like 3 or 5, but to get great results you'll want to use something larger (60-200 steps).
    • You can set the output_dir to the directory you wish to save to

Python Package

Setup

Install the package

pip install -U stable_diffusion_videos

Authenticate with Hugging Face

huggingface-cli login

Making Videos

Note: For Apple M1 architecture, use torch.float32 instead, as torch.float16 is not available on MPS.

from stable_diffusion_videos import StableDiffusionWalkPipeline
import torch

pipeline = StableDiffusionWalkPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    torch_dtype=torch.float16,
    revision="fp16",
).to("cuda")

video_path = pipeline.walk(
    prompts=['a cat', 'a dog'],
    seeds=[42, 1337],
    num_interpolation_steps=3,
    height=512,  # use multiples of 64 if > 512. Multiples of 8 if < 512.
    width=512,   # use multiples of 64 if > 512. Multiples of 8 if < 512.
    output_dir='dreams',        # Where images/videos will be saved
    name='animals_test',        # Subdirectory of output_dir where images/videos will be saved
    guidance_scale=8.5,         # Higher adheres to prompt more, lower lets model take the wheel
    num_inference_steps=50,     # Number of diffusion steps per image generated. 50 is good default
)

Making Music Videos

New! Music can be added to the video by providing a path to an audio file. The audio will inform the rate of interpolation so the videos move to the beat 🎶

from stable_diffusion_videos import StableDiffusionWalkPipeline
import torch

pipeline = StableDiffusionWalkPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    torch_dtype=torch.float16,
    revision="fp16",
).to("cuda")


# Seconds in the song.
audio_offsets = [146, 148]  # [Start, end]
fps = 30  # Use lower values for testing (5 or 10), higher values for better quality (30 or 60)

# Convert seconds to frames
num_interpolation_steps = [(b-a) * fps for a, b in zip(audio_offsets, audio_offsets[1:])]

video_path = pipeline.walk(
    prompts=['a cat', 'a dog'],
    seeds=[42, 1337],
    num_interpolation_steps=num_interpolation_steps,
    audio_filepath='audio.mp3',
    audio_start_sec=audio_offsets[0],
    fps=fps,
    height=512,  # use multiples of 64 if > 512. Multiples of 8 if < 512.
    width=512,   # use multiples of 64 if > 512. Multiples of 8 if < 512.
    output_dir='dreams',        # Where images/videos will be saved
    guidance_scale=7.5,         # Higher adheres to prompt more, lower lets model take the wheel
    num_inference_steps=50,     # Number of diffusion steps per image generated. 50 is good default
)

Run the App Locally

from stable_diffusion_videos import StableDiffusionWalkPipeline, Interface
import torch

pipeline = StableDiffusionWalkPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    torch_dtype=torch.float16,
    revision="fp16",
).to("cuda")

interface = Interface(pipeline)
interface.launch()

Credits

This work built off of a script shared by @karpathy. The script was modified to this gist, which was then updated/modified to this repo.

Contributing

You can file any issues/feature requests here

Enjoy 🤗

Extras

Upsample with Real-ESRGAN

You can also 4x upsample your images with Real-ESRGAN!

It's included when you pip install the latest version of stable-diffusion-videos!

You'll be able to use upsample=True in the walk function, like this:

pipeline.walk(['a cat', 'a dog'], [234, 345], upsample=True)

The above may cause you to run out of VRAM. No problem, you can do upsampling separately.

To upsample an individual image:

from stable_diffusion_videos import RealESRGANModel

model = RealESRGANModel.from_pretrained('nateraw/real-esrgan')
enhanced_image = model('your_file.jpg')

Or, to do a whole folder:

from stable_diffusion_videos import RealESRGANModel

model = RealESRGANModel.from_pretrained('nateraw/real-esrgan')
model.upsample_imagefolder('path/to/images/', 'path/to/output_dir')