Interactive Facial Animation: Enhancing Facial Rigs With Real-Time Shell And Contact Simulation

Demand for high-quality virtual 3D characters continues to grow in both entertainment and communication, and with it, the desire for interactive avatars. This presents unique challenges for high-fidelity interactive solutions that must be quick to set up, efficient to run in real-time on consumer hardware, and work well, not only on human characters, but also on fantastical and heavily stylized ones.  We present Interactive Facial Animation (IFA), a novel approach to augment facial animation rigs with shell simulation and contact without requiring anatomical priors or artist intervention. Our method uses an efficient shell model that tracks the output of a face rig in real-time and incorporates realistic contact. By designing our shell model to be differentiable, we can use inverse simulation to reconstruct realistic non-uniform stiffness that captures both stiff (ears, nose) and soft (cheeks, lips) areas, based on the rig’s deformation space. The reconstruction process imposes hard limits on acceptable errors, which ensures that IFA preserves the resolution and expressiveness of the source material, a common shortcoming in many face simulation pipelines. With performance in mind, we demonstrate IFA using a variant of the Fast Mass-Springs method. However, the presented concept can be realized with any differentiable physics solver that provides reliable convergence.  We showcase our method in four different character rigs, achieving simulation update runtimes between 0.39 and 1.27 milliseconds (787-2564 FPS) on a single CPU thread. Unlike existing methods that require lengthy preparation times, our pipeline can be ready to use within minutes, also on the CPU.