Logo Logo
directory schraeg
Camcorders· Cinema-Kamera· Computers· Drohnen· GPU· Kamera-Zubehör· Video-DSLR· accessories
Compositing· Color correction· DV Editing

Shooting· Sound
Forschung· Reviews· Streaming
/// News
Moving objects in videos can be masked only by description thanks to new AI

Moving objects in videos can be masked only by description thanks to new AI

[16:02 Tue,23.August 2022   by ]    

Masking moving objects in videos to edit them separately from all other content used to be an extremely time-consuming job. In the meantime, thanks to new DeepLearning algorithms, it has become easier and easier - often it is no longer even necessary to draw a mask around an object, but rather it is sufficient to set several points in order to select an object, which is then recognized by the algorithm on its own and tracked over the next frames and masked.

A horse that jumps high

The new Deep Learning algorithm ("Language as Queries for Referring Video Object Segmentation") from a team at the University of Hong Kong simplifies this work even further: here, the desired object can simply be described by text to select it such as "a horse jumping high jumps":

Based on this simple simple description, the object in the video is detected and tracked by a dynamic mask across all subsequent frames. The combined speech and image analysis of the algorithm works so well that the description of the desired object can also be quite complex. For example, the algorithm also identifies an object by its dynamic relationships to other objects or its location in space, such as "the person riding a skateboard."

To do this, the AI algorithm performs a whole series of complex tasks from the areas of text and video understanding: first it has to "understand" the input text, then it has to correctly recognize all objects in a video including dynamic relationships, and based on the description given by the user, it has to identify the correct object (including its properties, such as color, and relationships to another object, such as "the tennis racket in the hand of the player with the red shirt").


Then the object must be separated from the background and tracked across all frames in which it appears, forming a dynamic mask from it - even if the object changes its appearance through movement and changes in perspective. Ideally, the mask no longer needs to be adjusted by hand afterwards. Even dynamic actions that span a video sequence are correctly detected in a longer video, such as "the hand giving the dog a ball."

The new algorithm improves upon previous similar MEthodes once again and demonstrates very clearly the complex tasks that can now be accomplished by combining different Deep Learning methods. In its current state, the method could already be used to find specific objects in a video archive together with their relationship to other objects and to extract them. A small further step would, for example, also enable searching by natural speech input, and a larger further step could enable editing objects together with their exchange in a video by speech input.


As always, the corresponding porgram code can also be tried out by yourself - assuming the appropriate prior knowledge. If you want to learn more about the method, you can read the corresponding research paper.

Link more infos at bei github.com

deutsche Version dieser Seite: Bewegte Objekte in Videos können dank neuer KI nur per Beschreibung maskiert werden


  Vorige News lesen Nächste News lesen 
bildCanon: Kamerakühlung per magnetischer Flüssigkeit? bildYoloLiv Instream: erstes vertikales Multikamera-Live-Streaming-Studio

related news:1E0Whisper: New free AI turns speech into text and automatically translates into all languages 26.September 2022
VFX made easy: animate faces and fake voices with AI 22.September 2022
AI images with stable diffusion can now be used directly ni Photoshop and Gimp via plugin 18.September 2022
Generate and edit personalized images via AI 15.September 2022
Free image generation via AI: Stable Diffusion now with easy Windows Installer 13.September 2022
Insta360 X3 - 360° actioncam with new image sensors and KI functions 11.September 2022
How to produce an animated fashion show with free AI tools 2.September 2022
alle Newsmeldungen zum Thema Machine Learning

[nach oben]

Archiv Newsmeldungen


September - August - July - June - May - April - March - February - January

December - November - October - September - August - July - June - May - April - March - February - January






















deutsche Version dieser Seite: Bewegte Objekte in Videos können dank neuer KI nur per Beschreibung maskiert werden

last update : 28.September 2022 - 18:02 - slashCAM is a project by channelunit GmbH- mail : slashcam@--antispam:7465--slashcam.de - deutsche Version