latentsync

🔥 Updates

• 2025/06/11: LatentSync 1.6 is trained on 512 × 512 resolution videos to reduce softness in generated frames. Explore the latest demo through the workspace.
• 2025/03/14: LatentSync 1.5 improved temporal consistency with a temporal layer, improved performance on Chinese-language videos, and reduced stage-two training memory to 20 GB through a series of optimizations.

📖 Introduction

We present LatentSync, an end-to-end lip-sync method based on audio-conditioned latent diffusion models without an intermediate motion representation. Unlike earlier diffusion methods that work directly in pixel space or depend on two-stage generation, the framework uses the expressive capacity of Stable Diffusion to model complex audio-visual relationships.

🏗️ Framework

LatentSync uses Whisper to convert a melspectrogram into audio embeddings, which are integrated into the U-Net through cross-attention layers. Reference and masked frames are channel-wise concatenated with noised latents as the U-Net input. During training, a one-step method estimates clean latents from predicted noise before decoding them into frames. TREPA, LPIPS, and SyncNet losses are added in pixel space.

🎬 Demo

Original video	Lip-synced video
demo2_input.mp4	demo2_output_v1.6.mp4
demo3_input.mp4	demo3_output_v1.6.mp4
demo4_input.mp4	demo4_output_v1.6.mp4
demo5_input.mp4	demo5_output_v1.6.mp4
demo4_video.mp4	demo4_output.mp4

Photorealistic examples use contracted performers, while the animated examples are generated character footage.

📑 Open-source Plan

• Inference code and checkpoints
• Data processing pipeline
• Training code

🔧 Setting up the Environment

Install the required packages and download the checkpoints with:

source setup_env.sh

After the download completes, the checkpoint directory should look like this:

./checkpoints/
|-- latentsync_unet.pt
|-- whisper
|   `-- tiny.pt

You can also download latentsync_unet.pt and tiny.pt manually from the model workspace.

🚀 Inference

Minimum VRAM for inference:

• 8 GB with LatentSync 1.5
• 18 GB with LatentSync 1.6

There are two ways to perform inference:

1. Gradio App

Run the Gradio app for inference:

python gradio_app.py

2. Command Line Interface

Run the command-line script for inference:

./inference.sh

You can adjust the following inference parameters for different results:

• inference_steps [20-50]: higher values improve visual quality while increasing generation time.
• guidance_scale [1.0-3.0]: higher values can improve lip-sync accuracy, but may introduce distortion or jitter.

🔄 Data Processing Pipeline

The complete data processing pipeline includes the following steps:

1. Remove broken video files.
2. Resample video to 25 FPS and audio to 16000 Hz.
3. Detect scenes with PySceneDetect.
4. Split each video into segments of 5-10 seconds.
5. Affine-transform faces using detected landmarks, then resize them to 256 × 256.
6. Remove videos with a sync confidence score below 3 and set the audio-visual offset to 0.
7. Calculate the hyperIQA score and remove videos scoring below 40.

Run the processing script with:

./data_processing_pipeline.sh

Change the input_dir parameter to select the source data. Processed videos are written to the high_visual_quality directory, with each stage saved separately so interrupted runs can resume without repeating the full pipeline.

🏋️‍♂️ Training U-Net

Before training, process the data as described above. A pretrained SyncNet checkpoint is available for U-Net supervision, with reported accuracy on VoxCeleb2 and HDTF datasets.

huggingface-cli download ByteDance/LatentSync-1.6 stable_syncnet.pt --local-dir checkpoints

When preparation is complete, train the U-Net with:

./train_unet.sh

Several U-Net configurations are provided in configs/unet:

• stage1.yaml: Stage 1 training, requiring 23 GB VRAM.
• stage2.yaml: Stage 2 training with optimal performance, requiring 30 GB VRAM.
• stage2_efficient.yaml: Efficient Stage 2 training, requiring 20 GB VRAM, with a small possible tradeoff in visual quality and temporal consistency.
• stage1_512.yaml: Stage 1 training on 512 × 512 videos, requiring 30 GB VRAM.
• stage2_512.yaml: Stage 2 training on 512 × 512 videos, requiring 55 GB VRAM.

Update the U-Net configuration with the data directory, checkpoint path, and other training parameters. To create a data file list, run:

python -m tools.write_fileslist

🏋️‍♂️ Training SyncNet

To train SyncNet on your own dataset, run the following script. Its data processing pipeline matches the U-Net workflow:

./train_syncnet.sh

After validations_steps training, loss charts are saved to train_output_dir. They include both training and validation loss. Follow the architecture guide when customizing SyncNet for different image resolutions or input frame lengths.

📊 Evaluation

Evaluate the sync confidence score of a generated video with:

./eval/eval_sync_conf.sh

Evaluate SyncNet accuracy on a dataset with:

./eval/eval_syncnet_acc.sh

The released SyncNet is trained on data processed by the supplied pipeline, including affine transformation and audio-visual adjustment. Test data should go through the same preparation before evaluation.

🙏 Acknowledgement

• Our code is built on AnimateDiff.
• Some components are adapted from MuseTalk, StyleSync, SyncNet, and Wav2Lip.

We are grateful to the open-source community for these contributions.

📖 Citation

If you find the repository useful for your research, please cite the paper:

@article{li2024latentsync,
  title={LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision},
  author={Li, Chunyu and Zhang, Chao and Xu, Weikai and Lin, Jingyu and Xie, Jinghui and Feng, Weiguo},
  journal={arXiv preprint arXiv:2412.09262},
  year={2024}
}