Runtime MetaHuman Lip Sync (AI for NPCs) (+ CC4, Genesis, ARKit, and more)

Bring your MetaHuman and custom characters to life with zero-latency, real-time lip sync!

ℹ️ Note: The images with plugin examples and the demo project were created using the Runtime Audio Importer and/or Runtime Text To Speech / Runtime AI Chatbot Integrator plugins. So, to follow these examples, you will need to install these plugins as well. However, you can also implement your own audio input solution without using them.

• 🌟 Realistic Model - Enhanced visual fidelity specifically for MetaHuman characters with more natural mouth movements (250 morph target changes)

• ⚡ Standard Model - Broad compatibility with MetaHumans and custom characters (14 visemes)

Transform your digital characters with seamless, real-time lip synchronization that works completely offline and cross-platform! Watch as your characters respond naturally to speech input, creating immersive and believable conversations with minimal setup.

🚀 Key features:

• Real-time Lip Sync from microphone input and any other audio sources

• Dynamic laughter animations from detected audio cues

• Pixel Streaming microphone support - enable live lip sync from browser-based input!

• Offline Processing - no internet connection required

• Cross-platform Compatibility: Windows and Meta Quest

• Optimized for real-time performance on all platforms

• Works with both MetaHuman and custom characters:

  • Popular commercial characters (Daz Genesis 8/9, Reallusion CC3/CC4, Mixamo, ReadyPlayerMe)

  • FACS-based character models

  • ARKit blendshape standard

  • Any character with custom morph targets for facial expressions

• Multiple Audio Sources:

  • Live microphone input (via Runtime Audio Importer's capturable sound wave)

  • Captured audio playback (via Runtime Audio Importer's capturable sound wave)

  • Synthesized speech (via Runtime Text To Speech or Runtime AI Chatbot Integrator)

  • From audio file/buffer

  • Custom audio source (including streaming)

🎮 Perfect for:

• Interactive NPCs and digital humans

• Virtual assistants and guides

• Cutscene dialogue automation

• Live character performances

• VR/AR experiences

• Educational applications

• Accessibility solutions

🌟 Works great with:

• Runtime Audio Importer - For microphone capture and audio processing

• Runtime Text To Speech - For local (offline) text-to-speech

• Runtime AI Chatbot Integrator - For ElevenLabs and OpenAI text-to-speech

Technical Details

This plugin provides real-time lip synchronization for MetaHuman (as well as custom) characters by processing audio input to generate visemes.

The plugin relies on the onnxruntime library (MIT licensed) as the cross-platform native machine learning accelerator for lip sync inference. It leverages the NNERuntimeORT plugin that comes out-of-the-box in UE 5.6+ to provide onnxruntime functionality.

The plugin internally generates visemes (visual representations of phonemes) based on the audio input. The Standard Model produces 14 visemes and performs lip sync animation using a predefined pose asset. In contrast, the Realistic Model (exclusive to MetaHuman characters) generates 250 morph target changes without relying on a predefined pose asset, resulting in significantly more realistic facial animations.

Features:

• Simple, intuitive setup

• Real-time and offline viseme generation

• Universal language support - works with any spoken language through audio analysis

• Multiple audio input sources (microphone, playback, synthesized speech, such as ElevenLabs or OpenAI, custom PCM, Pixel Streaming)

• Direct integration with MetaHuman's face animation system

• Support for popular commercial character systems (Daz Genesis 8/9, Reallusion CC3/CC4, Mixamo, ReadyPlayerMe)

• Compatible with FACS-based blendshapes, ARKit standard, and custom morph targets

• Configurable interpolation settings

• Blueprint-friendly implementation

• No external dependencies or internet required