Logo Logo
directory schraeg
Camcorders· Cinema-Kamera· Computers· Drohnen· GPU· Kamera-Zubehör· Video-DSLR· accessories
Compositing· Color correction· DV Editing

Shooting· Sound
Forschung· KI· Reviews· Streaming
/// News
Video editing by text command: After text-to-image comes text-to-video

Video editing by text command: After text-to-image comes text-to-video

[16:25 Sun,11.September 2022   by ]    

An interesting video clip is making the rounds right now that may show the future of film production. The teaser, produced by Runway ML, demonstrates how videos can be edited and produced using text-only descriptions. It shows how a series of text inputs first import a dynamic street scene and give it a FilmLook, after which an object is roughly marked with a brush and seamlessly removed from the scene.

After that, you can see how an image of a "lush garden" is generated via text command - just as known from an AI image generator. However, via "slowly zoom in" a virtual camera zoom is then started and a text is slowly faded in. Very interesting is the third scene - here, as it seems, a person is masked from real footage by a simple "greenscreen character" command, then the background is blurred and finally desaturated.

This demonstrates various AI techniques that are already known: the generation of images, the targeted seamless removal of objects in a video, the generation of backgrounds, the masking of objects including keying - all through text commands.

The clip suggests that all these techniques also work in video editing, although it is of course not clear from the teaser how far the video-specific functions actually go, and whether the clip was really generated by the techniques mentioned or the targeted range of functions was only simulated via compositing. The big step towards AI-based generation of entire video sequences does not seem to have been taken yet, because the video sequence at the beginning is explicitly imported and not created.

However, since the clip comes from Runway ML, it can be assumed that the corresponding possibilities of text-controlled and object-based editing via AI will soon be available for video as well. Green screen, inpainting and motion tracking are already part of the functionality of Runway&s online video editor, so text commands would be the main new feature.

Online Video Editor interface

And even if the image quality and resolution should not be too high at first (stable diffusion and similar algorithms only work on computationally expensive detours with high-resolution images and often there are still - especially with humans - problems with the correct display of generated bodies), an absolute revolution in video production is just around the corner - faster than expected.

Runway ML (Machine Learning) has been offering the online video editor with numerous AI functions for several years now. Part of the team is, among others, the Heidelberg-based AI researcher Patrick Esser, who also played a major role in the development of the AI image generator Stable Diffusion, which was only recently made publicly available free of charge. Esser had also recently published a video in which the background of a video is dynamically exchanged via text description.

You can currently sign up for a closed beta phase to soon be able to test the text-to-video functions demonstrated in the clip in the online video editor. We are extremely excited.



Link more infos at bei runwayml.com

deutsche Version dieser Seite: Video-Editing per Textkommando: Nach Text-zu-Bild kommt Text-zu-Video KI


  Vorige News lesen Nächste News lesen 
bildLETsDOK: Ab Montag Dokumentarfilmfestival in 149 Kinos bildIBC 2022: Blackmagic`s neue Ultimatte 12 Keyer im slashCAM Videoüberblick

related news:1E0Tascam Portacapture X6 - mobile 6-track audio recorder with XLR and 32bit float introduced 30.January 2023
VEGAS 20 Update 2 brings numerous innovations plus discount promotion 24.January 2023
DZO Film completes affordable full-frame cine zoom series Catta Ace with 18-35mm T2.9 17.January 2023
Paint.NET 5.0 - free Windows photo editor now with GPU acceleration 12.January 2023
DaVinci Resolve for iPad: Professional video editing and color correction for free 22.December 2022
New sensors from Canon - 4.6K S-35 and 5.7K Full Frame with Global Shutter 20.December 2022
New firmware: DJI Avata FPV drone gets 10-bit, new frame rates and more 13.December 2022
alle Newsmeldungen zum Thema DV Editing
1E0DaVinci Resolve for iPad: Professional video editing and color correction for free 22.December 2022
Photoshop for videos? Pixelmator Pro 3.2 gets Video Editing / Compositing Layer 7.December 2022
Mobile editing app LumaFusion: New version 3.2 - and now also for Android 11.November 2022
LumaFusion v3.1 update - mobile editing app now offers video scopes 20.July 2022
Is AI Revolutionizing Photography? Object-based editing with the help of DALL-E 2 30.June 2022
Events @ Filmfest München: Series editing, studio tour and more 20.June 2022
Video editing via AI: Simply replace backgrounds or make objects disappear 21.January 2022
alle Newsmeldungen zum Thema Montage

[nach oben]

Archiv Newsmeldungen


February - January

December - November - October - September - August - July - June - May - April - March - February - January























deutsche Version dieser Seite: Video-Editing per Textkommando: Nach Text-zu-Bild kommt Text-zu-Video KI

last update : 2.Februar 2023 - 08:02 - slashCAM is a project by channelunit GmbH- mail : slashcam@--antispam:7465--slashcam.de - deutsche Version