Already well into the era of “fake news”, things might be about to get a whole lot murkier. Researchers at the University of Washington have shown how they can create video clips of Barack Obama by using audio from other speeches.
At the moment, the program works by taking audio files of a person speaking and then converting them into realistic mouth shapes that are subsequently grafted onto the head of that person in a pre-existing video. The algorithm uses machine learning from video already taken of a person to learn how they move their lips and other facial expressions in order to create as perfect a lip sync as is possible.
Because of these requirements, the researchers chose to trial their work on the former US President Barack Obama, as there are hours of videos of him talking freely available on the Internet. Previously, in order to generate the perfect mouth shape from audio, the process involved filming multiple people in a studio saying the same sentence over and over again. But this is obviously highly time-consuming and costly.
They managed to overcome this by first training a neural network to watch videos of Obama, and then taking the audio and converting it into different mouth shapes. The challenge was to deal with the “uncanny valley” problem that often occurs in which the synthesized humans look almost right, but just miss the mark and make those watching feel uncomfortable.
According to the researchers, people are particularly sensitive to any areas around the mouth that are not quite right. They were able to create a new mouth synthesis technique, in which they could then realistically superimpose and blend the computer-generated mouths onto real videos of Obama talking about other subjects. Crucially, they inserted a short time shift, giving the neural network time to anticipate what Obama was going to say next.
They were careful to only use audio from the same person that they were simulating so as not to put words in someone else’s mouth, but unfortunately, it is unlikely to stay this way. The researchers have ideas of using the technology for conference meetings, or even perhaps to create VR of dead historical figures.
While these aims of the researchers may be noble, I think we can all see the direction that the tech is likely to take. With the media and Internet abuzz with “fake news”, such programs and algorithms are ripe for abuse, as people will be able to make videos of politicians or celebrities “saying” things that they obviously never have. It is sure to make fact-checking and truth a whole lot murkier as the technology becomes more established.
1