Logo Logo
directory schraeg
Camcorders· Cinema-Kamera· Computers· Drohnen· GPU· Kamera-Zubehör· Video-DSLR· accessories
Compositing· Color correction· DV Editing

Shooting· Sound
Forschung· Reviews· Streaming
/// News
Video editing by text command: After text-to-image comes text-to-video

Video editing by text command: After text-to-image comes text-to-video

[16:25 Sun,11.September 2022   by ]    

An interesting video clip is making the rounds right now that may show the future of film production. The teaser, produced by Runway ML, demonstrates how videos can be edited and produced using text-only descriptions. It shows how a series of text inputs first import a dynamic street scene and give it a FilmLook, after which an object is roughly marked with a brush and seamlessly removed from the scene.

After that, you can see how an image of a "lush garden" is generated via text command - just as known from an AI image generator. However, via "slowly zoom in" a virtual camera zoom is then started and a text is slowly faded in. Very interesting is the third scene - here, as it seems, a person is masked from real footage by a simple "greenscreen character" command, then the background is blurred and finally desaturated.

This demonstrates various AI techniques that are already known: the generation of images, the targeted seamless removal of objects in a video, the generation of backgrounds, the masking of objects including keying - all through text commands.

The clip suggests that all these techniques also work in video editing, although it is of course not clear from the teaser how far the video-specific functions actually go, and whether the clip was really generated by the techniques mentioned or the targeted range of functions was only simulated via compositing. The big step towards AI-based generation of entire video sequences does not seem to have been taken yet, because the video sequence at the beginning is explicitly imported and not created.

However, since the clip comes from Runway ML, it can be assumed that the corresponding possibilities of text-controlled and object-based editing via AI will soon be available for video as well. Green screen, inpainting and motion tracking are already part of the functionality of Runway&s online video editor, so text commands would be the main new feature.

Online Video Editor interface

And even if the image quality and resolution should not be too high at first (stable diffusion and similar algorithms only work on computationally expensive detours with high-resolution images and often there are still - especially with humans - problems with the correct display of generated bodies), an absolute revolution in video production is just around the corner - faster than expected.

Runway ML (Machine Learning) has been offering the online video editor with numerous AI functions for several years now. Part of the team is, among others, the Heidelberg-based AI researcher Patrick Esser, who also played a major role in the development of the AI image generator Stable Diffusion, which was only recently made publicly available free of charge. Esser had also recently published a video in which the background of a video is dynamically exchanged via text description.

You can currently sign up for a closed beta phase to soon be able to test the text-to-video functions demonstrated in the clip in the online video editor. We are extremely excited.



Link more infos at bei runwayml.com

deutsche Version dieser Seite: Video-Editing per Textkommando: Nach Text-zu-Bild kommt Text-zu-Video KI


  Vorige News lesen Nächste News lesen 
bildLETsDOK: Ab Montag Dokumentarfilmfestival in 149 Kinos bildIBC 2022: Blackmagic`s neue Ultimatte 12 Keyer im slashCAM Videoüberblick

related news:1E0New Boris FX Plugin units for Final Cut Pro cost 59 Dollar 19.September 2022
IBC 2022: Frame.io and Camera to Cloud use cases 15.September 2022
IBC 2022: Sony's new FR7 full-frame PTZ in a slashCAM video overview 13.September 2022
Sony FR7 - first PTZ camera with full frame sensor and interchangeable lenses 6.September 2022
Sirui introduces two more full frame 1.6x anamorphic lenses and an adapter 29.August 2022
Tamron announces 20-40mm F/2.8 Di III VXD full-frame zoom lens 18.August 2022
Atomos Connect now supports Frame.io Camera-to-Cloud 12.August 2022
alle Newsmeldungen zum Thema DV Editing
1E0LumaFusion v3.1 update - mobile editing app now offers video scopes 20.July 2022
Is AI Revolutionizing Photography? Object-based editing with the help of DALL-E 2 30.June 2022
Events @ Filmfest München: Series editing, studio tour and more 20.June 2022
Video editing via AI: Simply replace backgrounds or make objects disappear 21.January 2022
Audio Design Desk 1.9: Real-time editing tool for sound effects now with new features and Lite versi 15.January 2022
Hedge Postlab for Media Composer enables team editing in the cloud for Avid editors 6.November 2021
Sony C3 Portal: Cloud solution for Sony cameras - from shooting directly to the editing suite 23.September 2021
alle Newsmeldungen zum Thema Montage

[nach oben]

Archiv Newsmeldungen


September - August - July - June - May - April - March - February - January

December - November - October - September - August - July - June - May - April - March - February - January






















deutsche Version dieser Seite: Video-Editing per Textkommando: Nach Text-zu-Bild kommt Text-zu-Video KI

last update : 28.September 2022 - 18:02 - slashCAM is a project by channelunit GmbH- mail : slashcam@--antispam:7465--slashcam.de - deutsche Version