New Google Dreamix AI Demonstrates Video Editing of the Future

[12:27 Fri,10.February 2023 by Thomas Richter]

Almost simultaneously with Runway&s Gen1 video AI, Google has introduced a similar AI. Google&s calls Dreamix an AI video editor and emphasizes its capabilities to not only freely generate videos via text prompt but also modify targeted objects videos. Dreamix offers three methods to generate videos: subject-driven video generation, video editing and image-to-video.

Using Subject Driven Video Generation, an object can be animated by 10-20 photos per prompt. In the video example, four pictures of a teddy bear and the description "a walking bear" are used to create a short clip in which the teddy bear seen in the photos is walking. Color, posture, object size, image composition and camera position from the source video are kept. However, it is also possible to simply replace one object with another without affecting the rest of the scene.

In Video Editing mode, a video serves as input - text prompts can then be used to manipulate specific elements. Thus, the movement or texture of an object appearing in the video can be changed, as well as the background. Objects can also be added in a meaningful way.

Likewise, a single photo can be used as a template (image-to-video) and an object can be animated from it in a video via prompt or new objects can be added. Camera movements are also possible in the new scene or a subsequent time-lapse recording.

Google uses a video diffusion model for Dreamix, which is a similar method to current image AIs such as DALL-E2 or the open-source Stable Diffusion. This involves heavily downscaling an input video, adding artificial noise, and then processing it in a video diffusion model, which then uses a text prompt to generate a new video from it that retains some properties of the original video and re-renders others according to the text input.

Dreamix Model

Runway&s

just unveiled Video AI Gen 1 for its online AI Viceo Editor had primarily demonstrated the ability to change textures of videos via text-driven style transfer, but a

demo released in September had also already demonstrated capabilities of object-based video editing, such as masking objects via text command - Google has demonstrated even more capabilities with Dreamix.

Object-based editing: making complex work simple

.
In the very near future, such video AI promise numerous extremely useful functions and promise a revolution for working with video. For example, with the help of video AIs, individual objects in videos can be selectively masked or manipulated using only a text prompt, and VFX can be added to a scene. Everything only by text prompt (or soon presumably also by voice command) completely without sophisticated compositing. Also, with such AIs, animations of objects could be produced simply by image template and text description. These universal capabilities would replace a large number of previous special functions in compositing and video editing programs with simple text prompts, thus greatly simplifying and shortening previously complex work.

more infos at bei dreamix-video-editing.github.io

deutsche Version dieser Seite: Google Dreamix: Neue KI demonstriert die Videobearbeitung der Zukunft