The model configuration that is publicly available is about 5x slower than real-time (~6fps). At lower resolution and with a less conservative number of diffusion steps, we are able to generate the video at 20-23 fps, which is just about real-time. Here is an example: https://6ammc3n5zzf5ljnz.public.blob.vercel-storage.com/fast...

We use rectified flow for denoising, which is a (relatively) recent advancement in diffusion models that allow them to run a lot faster. We also use a 3D VAE that compresses the video along both spatial and temporal dimensions. Temporal compression also improves speed.

Checkout Tavus.io for realtime. They have a great API for realtime conversational replicas. You can configure the CVI to do just about anything you want to do with a realtime streaming replica.