Snap Unveils Real-Time Image Model for Augmented Reality Experiences

On Tuesday at the Augmented World Expo, Snap showcased an initial version of its real-time, on-device image diffusion model, designed to create dynamic augmented reality (AR) experiences. The event also revealed new generative AI tools aimed at empowering AR creators.

Speaking at the expo, Snap’s co-founder and CTO Bobby Murphy revealed that the company’s new model is compact enough to operate on a smartphone. He emphasized its capability of re-rendering frames instantly in response to text prompts.

He also highlighted the excitement surrounding generative AI image diffusion models but noted that their current pace is insufficient for practical augmented reality applications. To address this issue, Snap’s teams have been focusing their efforts on enhancing the performance of machine learning models to improve speed.

In the upcoming months, Snapchat users can expect to see Lenses powered by this new generative model. The company intends to make the tool available to creators by the year’s end.

Murphy said, “This and future real-time on-device generative ML models speak to an exciting new direction for augmented reality, and is giving us space to reconsider how we imagine rendering and creating AR experiences altogether.”

At yesterday’s expo, Murphy also announced the release of Lens Studio 5.0. This new version, aimed at developers, includes advanced generative AI tools designed to significantly speed up the creation of AR effects, potentially reducing development time from months to weeks.

Equipped with advanced tools, AR creators can craft selfie Lenses featuring lifelike machine-learning face effects. They can also develop custom stylization effects that dynamically transform the user’s face, body, and surroundings in real time. Additionally, creators have the capability with this tool to generate 3D assets within minutes and develop characters with image or text prompts using Snap’s Face Mesh technology.