Sign in with your favourite social login to share, comment and pin your favourites.
New video software can put a stranger’s words in your mouth
Now it’s possible for someone to literally put words in your mouth. Not only that, they can make you smile and frown too. That’s thanks to a new piece of video software called facial expression re-enactment, which has been developed by researchers from the University of Erlangen-Nuremberg, the Max Planck Institute for Informatics and Stanford University.
It’s extremely clever. The source subject – i.e. the person whose expressions and lip movements you want to take – is filmed using a camera that encodes colour and depth data, such as the Microsoft Kinect. This ensures that their skin colour, shape, head pose and movements are all recorded accurately. The software then generates a proxy mouth using a generic image that fits closely to the target’s face, adding shadows and other lighting effects, and blends a digital representation of the facial movements onto the target’s face, adjusting for skin colour, head movements, head shape and lighting.
And there you go – it looks like the target is acting exactly like the source.
The video software works in real time, so could happen over a live video call, and accounts for dynamic skin detail like wrinkles.
The only thing it can’t do is match the voice. Not yet, anyway.
The team behind it say it could be used in translation during multi-lingual video conferences, as well as in virtual reality worlds and video editing. They’re aware of the possible misuses, such as making it appear that someone said something they didn’t. In fact, they tell the Wall Street Journal, it could be used to detect when video has been manipulated.
“It’s important to make people aware of it and give people the technology so that they can detect manipulations,” said team member Matthias Niessner.
They will present the technology at a computer graphics conference in Kobe, Japan. They are also planning to consult with tech industry experts to see if they can make a business plan.