By: Nick Gambino
Dubbing movies and TV for a foreign language audience has never been perfect. In the 70s and 80s, the lack of continuity between the mouth of the actor and the words spoken was a running joke. Check out any Kung Fu flick from that era and you’ll see what I mean.
In recent years, they’ve improved the dub so at least there’s some connective tissue between the audio and the visual. But despite best efforts, as long as the actor’s mouth looks like it’s saying something else, there will never be a seamless dub.
AI startup, Flawless, is looking to change that with the use of advanced deepfake technology. By focusing on the actor’s mouth, they’re able to create the illusion of said actor speaking in a different language.
All studios or production companies need to do is input the movie or TV show into the Flawless machine learning model alongside the dubbed language recording and Flawless will take care of the rest. The AI creates lip movements that correlate with the language recording and then attach those lips to the actor.
The way I’ve described it sounds hokey, much like those old Conan O’Brien gags where he’d swap out Schwarzenegger’s mouth with an impersonator. But using deepfake is a lot more convincing than those crude attempts.
“When someone’s watching this dubbed footage, they’re not jolted out of the performance by a jarring word or a mistimed mouth movement,” Nick Lynes, co-founder of Flawless told The Verge. “It’s all about retaining the performance and retaining the original style.”
Their demo reel is pretty convincing though not flawless. Messing with the actor’s face without being too distracting is a difficult task. I am personally not a big fan of dubbing or anything that alters the actor’s performance, specifically in non-CG heavy movies. I watch foreign language films all the time with subtitles and much prefer it. I’m aware that for accessibility reasons, it’s important to provide dubbed versions of movies, but I’m not sure altering their mouths falls under the category of “accessibility.”
This “ground-breaking neural net technology” has yet to prove itself, but they’re young (the company itself only launched a month ago) and if they can wow us with deepfake dubs the length of a feature film, maybe they’ll become the future of foreign language cinema consumption.
