DeepFaceVideoEditing: AI enables changing facial expressions in videos via sketching

[11:38 Thu,11.August 2022 by Thomas Richter]

Another step on the way to object-based video editing via AI has been taken by a team of Chinese researchers accomplished - their algorithm, named "DeepFaceVideoEditing" and based on DeepLearning methods, enables the manipulation of facial expressions or even other features by means of a simple sketch of the desired change.

Changes to hairstyle, beard and eyebrows via DeepFaceVideoEditing

Users can select multiple frames from a video where a face is prominent and manually define a rough mask in the face in which the desired change is sketched. This change can affect the expression of the eyes or mouth as well as features such as a beard, hairstyle or eyebrows. Two fundamentally different types of changes can be realized: either part of the appearance of a face can be changed consistently over the length of the entire video (such as the hairstyle or a beard), or the change can affect only a small sequence of frames, such as a momentary change in facial expression like a smile or a brief raise of the eyebrows or sniffle.

In the future, video editors will be able to use DeepFaceVideoEditing (or similar methods that are still being developed and will hopefully be integrated into editing tools) to change the facial expressions or appearance of actors much more easily than before (e.g. by rotoscoping) (or to remove or add a tattoo or scar, for example). This opens up a whole new range of options in post-production, meaning that even serious artistic decisions can be made after the shoot and implemented with relative ease.

DeepFaceVideoEditing algorithm

DeepFaceVideoEditing uses, among other things, the

StyleGAN3 algorithm developed by Nvidia, which specializes in the generation and also manipulation of human faces. It is known, among other things, for the online project

Thispersondoesnotexist, which can be used to generate arbitrarily realistic-looking faces. And also

this algorithm for artificially aging faces by photo, which can also be

tried out online, uses StyleGAN.

Changing the Minim and the Hairstyle via DeepFaceVideoEditing

Since similar algorithms for editing faces in video are basically very similar to those for manipulating photos, but are again significantly more complex due to the large number of slightly different individual images, experience shows that they always take a little longer to develop. In addition, they require more computing power due to this higher complexity. But as the new algorithm shows, these problems can be overcome, so it is only a matter of time before (AI) algorithms that work for photos also work with videos.

more infos at bei geometrylearning.com

deutsche Version dieser Seite: DeepFaceVideoEditing: KI ermöglicht das Ändern von Gesichtsausdrücken in Videos