Omnimatte: Nearly perfect masks of moving objects via AI

[12:14 Fri,27.August 2021 by Thomas Richter]

As early as last year, a deep-learning algorithm developed by researchers at Google and Oxford University achieved astonishing results in recognizing moving objects in a video and manipulating them in a targeted manner. For example, people could be removed from a video almost without a trace, or the timing of their movements could be changed.

A new version of the algorithm from the same team (plus researchers from Israel&s Weizmann Institute of Science) now delivers even better results because it detects not only the objects but also their secondary effects, such as shadows or reflections. This means that objects can even be removed from moving videos together with the complex dynamic effects they cause, such as waves and clouds of dust or smoke.

Starting from an input video ("Original" in the image above) and rough masks of the objects ("Input mask"), the algorithm creates an omnimatte - an alpha matte "Omnimatte (alpha)" including the foreground color "Omnimatte (RGBA)", which contains the person itself and all scene elements associated with it, as well as the background ("Background") without the selected objects and their secondary tracks in the image.

The method allows very easy creation of moving masks/layers of multiple objects together with their correlating tracks in the image and their simple manipulation (such as by retiming) in a compositing program. The possibilities of intelligent, object-based video editing and compositing by such an algorithm are enormous: for example, for the purpose of a better image composition, the movements of objects in a video can be completely changed afterwards, objects can be removed from the video or multiplied (stroboscope effect) or, for example, the color can be changed selectively.

Stroboscope effect via Omnimatte

How well the new method works can be seen on the following page

Here are numerous videos with different motifs from which moving objects were removed. The study itself can be downloaded

here - and here the

program code.

As always, the new method is not yet perfect and also the resolution of the videos is still very low due to the high computing power required - but it shows the fast progress of on.

Here is the presentation of the "Ominimatte" method (also in comparison to the predecessor) by Two Minute Papers:

more infos at bei omnimatte.github.io

deutsche Version dieser Seite: Omnimatte: Nahezu perfekte Masken bewegter Objekte per KI