Logo Logo
directory schraeg
Camcorders· Cinema-Kamera· Computers· Drohnen· GPU· Kamera-Zubehör· Video-DSLR· accessories
Compositing· Color correction· DV Editing

Shooting· Sound
Forschung· Reviews· Streaming
/// News
Visual dubbing by AI makes every actor a native speaker

Visual dubbing by AI makes every actor a native speaker

[09:20 tue,25.May 2021   by ]    

The London-based startup Flawless AI wants to make dubbed films look better by using AI to adapt the lip movements of the person speaking to the text of the voice actor. We had already reported last year about the DeepLearning algorithm "Wav2Lip" of an Indian research team reported, which was able to match the lip movements of a speaker to the words from any audio recording in real time, this worked quite universally with any face, any language and any voice.

Similarly, Flawless AI&s TrueSync christened DeepFake Algorithm does this for mouth movements as the world&s first commercial system - which aims to enable filmmakers to visually translate foreign language films into the native language of any audience.


Flawless AI&s True Sync Algorithm

According to its own claims, it works fororealistically and lip sync is so seamless that the actor&s particular performance is preserved in detail and viewers are no longer distracted by inconsistencies between lip movements and what is spoken.

Demo of Flawless AI&s True Sync algorithm:

For this purpose, the algorithm, which is based on a research paper from 2019 and was developed in collaboration with the Max Planck Institute, analyzes all the mouth movements an actor makes in the original film while uttering different sounds, in order to subsequently synthesize them for the foreign language version with the dubbed text and merge them with the rest of the face as seamlessly as possible. To us, the result doesn&t seem quite perfect yet, but an improved version of the TrueSync algorithm is probably already in the works. Automation is not yet complete; while 85% of the visual dubbing works automatically, 15% of the work still has to be done manually.

Interesting for major movie studios, Netflix and more


Flawless AI, as can already be seen in the demo video, is targeting major film and series productions and is working with film studios such as Paramount, with Flawless AI&s co-founder, filmmaker Scott Mann, who, among other things, made the 2015 film "Heist" With Robert De Niro, providing the contacts. The first major film to use the method is scheduled for release in about a year.

In addition to the major film studios, a lip-sync algorithm would also be highly interesting for streaming providers such as Netflix or Amazon Prime, which could then automatically (and relatively inexpensively) visually adapt the many foreign-language versions of their films and series to the respective spoken language. Of course, a lip-sync algorithm would also be interesting on a smaller scale, such as for filmed lectures, press conferences, or animated films in other languages.


In addition to filmmakers and viewers, translators of dubbed texts would also benefit, as their work would be made much easier if they no longer had to ensure that the spoken text of the translation did not deviate too much from the lip movements of the original. This is because the dialog is often heavily altered during dubbing in order to better adapt it to the lip movements of the actors, which, however, can distort its meaning very much. However, even with lip-syncing via Ki, the length of the text or the timing would of course have to be exactly right.

For filmmakers, the technology also offers a simple alternative to costly reshoots when the spoken text of a scene needs to be changed after the fact.

Automatic foreign language versions on YouTube?

In the future, it would also be conceivable to offer other-language versions of any clips on YouTube, for example, completely automatically. After all, YouTube already provides an automatic transcription including subtitles, and the next steps are also already possible with the help of various Deep Learning algorithms: translating the transcribed text into another language, speech synthesis with the voice of the original, and then lip-syncing the video with the new audio.


In theory, however, the technique can of course be abused to generate clips in which people realistically appear to say things they never said - the new audio can also be generated by neural network to mimic the real voice.

Link more infos at bei www.flawlessai.com

deutsche Version dieser Seite: Visuelles Dubbing per KI macht jeden Schauspieler zum Mehrfach-Muttersprachler


  Vorige News lesen Nächste News lesen 
bildDJI-Konkurrenz: Neue 4K 60p Footage von Sony´s Airpeak Drohne mit montierter A7S III bildPanasonics neue GH Serie: Mehr Infos heute um 16:00 …

related news:1E0SmallRig Forevala L20: Affordable lavalier mic 25.September 2021
AI separates music into separate vocal and accompaniment audio tracks 20.September 2021
Roland V-02HD MK II Streaming Video Mixer 19.September 2021
Vimeo now also supports Dolby Vision HDR 13.September 2021
Tascam announces XLR audio adapters for Canon, Nikon and Fujifilm DSLMs 19.August 2021
LumaFusion 3.0 is here: video stabilization, audio EQ and support for external drives 31.July 2021
Sound Devices A20-Mini: Professional wireless audio transmitter with 32-bit float recording 23.July 2021
alle Newsmeldungen zum Thema Sound
1E0AI separates music into separate vocal and accompaniment audio tracks 20.September 2021
Perfect for music videos: Spatial object-based video editing via AI and 16 cameras 19.September 2021
Nvidia Broadcast App 1.3: Better live streaming and video conferencing via AI 6.September 2021
Omnimatte: Nearly perfect masks of moving objects via AI 27.August 2021
Per DeepFake itself star in the movie trailer of 22.August 2021
Alias-Free GAN: Nvidia's AI creates even more realistic animations of faces 27.July 2021
New Parrot Anafi Ai drone can be flown via LTE 19.July 2021
alle Newsmeldungen zum Thema Machine Learning

[nach oben]

Archiv Newsmeldungen


September - August - July - June - May - April - March - February - January

December - November - October - September - August - July - June - May - April - March - February - January





















deutsche Version dieser Seite: Visuelles Dubbing per KI macht jeden Schauspieler zum Mehrfach-Muttersprachler

last update : 27.September 2021 - 16:27 - slashCAM is a project by channelunit GmbH- mail : slashcam@--antispam:7465--slashcam.de - deutsche Version