Install Stable Video 4D (SV4D) Locally for Video to Video Generation

 Stable Video 4D (SV4D) is a generative model based on Stable Video Diffusion (SVD) and Stable Video 3D (SV3D), which takes in a single-view video of an object and generates multiple novel-view videos (4D image matrix) of that object.

Code:

conda create -n sdv4 python=3.11 -y && conda activate sdv4

conda install -c conda-forge nccl conda install pytorch torchvision torchaudio -c pytorch

git clone https://github.com/Stability-AI/generative-models.git && cd generative-models

pip install accelerate huggingface_hub

pip install black==23.7.0

pip install chardet==5.1.0

pip install clip@git+https://github.com/openai/CLIP.git

pip install einops>=0.6.1

pip install fairscale>=0.4.13

pip install fire>=0.5.0

pip install fsspec>=2023.6.0

pip install invisible-watermark>=0.2.0

pip install kornia==0.6.9

pip install matplotlib>=3.7.2

pip install natsort>=8.4.0

pip install ninja>=1.11.1

pip install numpy>=1.24.4

pip install omegaconf>=2.3.0

pip install open-clip-torch>=2.20.0

pip install opencv-python==4.6.0.66

pip install pandas>=2.0.3

pip install pillow>=9.5.0

pip install pudb>=2022.1.3

pip install pytorch-lightning==2.0.1

pip install pyyaml>=6.0.1

pip install rembg

pip install scipy>=1.10.1

pip install streamlit>=0.73.1

pip install tensorboardx==2.6

pip install timm>=0.9.2

pip install tokenizers

pip install torchdata==0.6.1

pip install torchmetrics>=1.0.1

pip install tqdm>=4.65.0

pip install transformers==4.19.1

pip install triton==2.0.0

pip install urllib3<1.27,>=1.25.4

pip install wandb>=0.15.6

pip install webdataset>=0.2.33

pip install wheel>=0.41.0

pip install xformers>=0.0.20

pip install gradio

pip install streamlit-keyup==0.2.0

pip install imageio==2.19.3 pip install imageio-ffmpeg==0.4.7

huggingface-cli login   # get the key from huggingface.co

export HF_TOKEN=""

wget --header="Authorization: Bearer $HF_TOKEN" -O checkpoints/sv4d.safetensors https://huggingface.co/stabilityai/sv4d/resolve/main/sv4d.safetensors

wget --header="Authorization: Bearer $HF_TOKEN" -O checkpoints/sv3d_u.safetensors https://huggingface.co/stabilityai/sv3d/resolve/main/sv3d_u.safetensors

ls -lh checkpoints/*.safetensors

python3 scripts/sampling/simple_video_sample_4d.py --input_path assets/test_video1.mp4 --output_folder outputs/sv4d