Microsoft Research - GASP: Gaussian Avatars with Synthetic Priors
The video introduces GASP, a new method for generating highly realistic, real-time animatable avatars with full 360-degree rendering capabilities. Unlike previous models that require multiple cameras, GASP can be trained using data from a single camera, making it accessible to average users. The method addresses the limitations of existing Gaussian avatar techniques, which struggle with novel view synthesis when trained with single-camera data. GASP uses a generative prior trained on a large synthetic dataset to fill in missing data, such as the sides and back of the head, which are typically not captured by a single camera. This synthetic dataset provides perfectly accurate labels and correspondence to a 3D morphable model (3DMM), overcoming the challenges of real-world datasets that often lack diversity and have imperfect annotations.
The training process involves using an auto-decoder where each subject is assigned a latent vector. The prior network maps these vectors, along with camera and 3DMM parameters, to obtain Gaussian avatar parameters. The model can then fit an avatar for a user using a single image or short video. The fitting process includes optimizing a latent code, fine-tuning the decoder to match unseen regions, and adjusting Gaussians to improve quality in available data areas. GASP is compared to state-of-the-art models, demonstrating superior performance in real-time animation and single-image avatar creation, while also being more computationally efficient than models like CFA, which require more resources and produce static expressions.
Key Points:
- GASP enables realistic avatar creation using single-camera data, making it accessible for average users.
- The method uses a generative prior trained on synthetic data to fill in gaps left by single-camera setups.
- GASP's training process involves an auto-decoder and optimization of latent codes to fit user-specific avatars.
- The model outperforms existing techniques in real-time animation and efficiency, running on a single GPU.
- GASP provides a significant improvement over models like CFA, which are resource-intensive and less flexible.