New Disney AI can age - or rejuvenate - actors in seconds

[11:43 Wed,14.December 2022 by Thomas Richter]

Disney has long been exploring the possibilities of using AIs in the form of neural networks to speed up otherwise labor-intensive and time-consuming processes in film (post)production. Now, a team of researchers from Disney&s and ETH Zurich has unveiled an algorithm that simply allows the age of actors to be changed, a task often required in films - the best-known example here is probably Martin Scorsese&s The Irishman (2019), in which Robert De Niro and Al Pacino were digitally rejuvenated.

The algorithm, christened FRAN (Face Re-Aging Network), can both artificially age (aging) and rejuvenate (de-aging) faces. It provides an alternative to the otherwise labor-intensive process of modifying individual images of a face by hand. FRAN also allows extensive artistic control over the result using simple and intuitive mechanisms that allow fine-tuning of the de-aging effect - a feature of great importance in real-world film production pipelines.

The following example shows several video frames of a person (age 35) aged or rejuvenated to 65 (top row) and 18 (bottom row).

De-aging ripe for film production

.
According to Disney, FRAN is the first fully automated de-aging algorithm ready for production - previous algorithms have had problems maintaining the identity of faces during artificial aging, as well as the consistency of changes during image sequences, or even generating images with a high enough resolution for movies. FRAN, on the other hand, delivers temporally consistent results for videos showing faces from arbitrary angles, under a wide range of lighting conditions and in motion, according to Disney.

This was made possible by a novel approach to training. For example, FRAN&s neural network was trained with a large dataset of pairs of randomly generated synthetic faces of different ages. This circumvented the need to find thousands of images of real people of different ages with the same facial expression, pose, lighting, and background. This allowed for higher quality results than previously possible. Here is the study

High-Resolution Neural Face Swapping for Visual Effects.

De-aging by FRAN and mask.

Limitations

.
But even the new de-aging algorithm is not perfect: FRAN is not good at making very large changes to age, for example, turning an adult&s face into a child&s - nor can it simulate the graying of hair. So, since manual VFX work and even the practical application of prosthetic makeup are not subject to these limitations, FRAN is unlikely to replace many jobs in the industry anytime soon. As can be seen in the second example, FRAN can be used well to augment age changes in the mask.

However, as is always the case with AI research and algorithms, the results are very likely to see significant improvement in the next study or two.

Disney has been researching VFX AIs for some time

.
Disney Research has been working on AI algorithms that can perform film post-production tasks for quite some time. For example, Disney had already presented a research paper together with ETH Zurich in July 2020, in which a

high-resolution DeepFake method was presented to exchange faces in videos and images fully automatically via neural network. With its help, the face of an actor could be replaced by that of any other person and deceptively animated by the latter.

Disney DeepFake

Another Disney algorithm, unveiled in December 2021, enabled

a realistic change in the lighting of generated faces complete with eyes, hair, and mouth with correct shadows cast - as well as more complex ambient lighting changes in direction and color. And, most importantly for Disney&s intended film role, the results also look realistic when animated - for example, when changing from one facial expression to another or when speaking.

more infos at bei studios.disneyresearch.com

deutsche Version dieser Seite: Neue Disney KI kann Schauspieler in Sekunden altern lassen - oder verjüngen