Over the last five years, AI has seen rapid improvements in its ability to generate synthetic versions of people’s faces and voices – commonly known as deepfakes. Generating the earliest deepfakes required powerful computers and technical expertise, but deepfake creation is now being increasingly democratised via intuitive interfaces and off-device processing that require no special skills or computing power.
This has led to the rise of deepfake apps, where anyone can create a deepfake from their smartphone. Some of these apps allow users to accurately swap their face with that of a celebrity, while others recreate a user’s facial movements in a video of another person. An app called Jiggy even generates GIFs of the user doing different dances based on a single photo of their body.
Deepfakes generated by these apps aren’t highly realistic, but more sophisticated versions of the technology have been used to create photorealistic images of non-existent people, replicate Jay Z’s voice “rapping” Shakespeare, and synchronise Snoop Dogg’s lip movements to Just Eat adverts.
Creating these realistic deepfakes still requires significant expertise and hardware. South Park creators Matt Stone and Trey Parker employed 20 professional “deepfakers” and VFX specialists to create a new satirical deepfake show, Sassy Justice, which follows a small-town TV reporter with Donald Trump’s face as he encounters deepfaked celebrities including Al Gore, Ivanka Trump and Michael Caine. Stone and Parker stated that the first 15-minute video had cost millions to produce.
But technical expertise and financial resources may not be barriers to creating realistic deepfakes for much longer, as developers race to harness deepfakes’ potential to define a new generation of social content. One company leading the way is Ukraine-based Reface, whose face-swapping app had been downloaded over 20 million times by mid-August 2020, topping both the Apple and Android stores in 100 countries. Reface’s CEO, Roman Mogylnyi, recently told TechCrunch that upgrades are in the pipeline to enhance the app’s deepfake quality and include full-body swaps. In Mogylnyi’s mind, the future of deepfake apps represents “a personalisation platform where people will be able to live different lives during their one lifetime.” But the commodification of advanced deepfake apps raises questions about how they could be misused.
Glimpses of this misuse are already visible. One of this article’s co-authors recently discovered a “deepfake pornography bot” on the messaging app Telegram, which allowed users to upload pictures of clothed women and “strip” them by generating their deepfake nude images. Over 100,000 of deepfake images of women and minors were shared on Telegram channels counting over 100,000 members.
Fears that deepfake apps could fuel the problem of political disinformation and deceptive content online were also sparked in April 2020, when Donald Trump retweeted a crudely manipulated video of Joe Biden lolling his tongue and twitching his eyebrows. Although the video wasn't realistic, similar scenarios in the future may be more convincing. Both examples point to a worrying future where deepfake apps could create harmful fakes on a massive scale, threatening anyone whose images are online.
Many deepfake apps address these concerns by being “on rails”, or restricted: users can only swap faces into a selection of scenes from pre-approved films or shows. But these restrictions are often the outcome of technological limitations rather than a deliberate security choice. In order to quickly generate high-quality face-swaps with one or a few user images, apps “pre-train” their generative models on a number of popular movie scenes, such as the twins from The Shining, or Sean Bean’s “one does not simply walk into Mordor” meme from The Lord of the Rings. As the technology becomes more powerful and pre-training less restrictive, developers might see a competitive advantage in opening up their apps to user-uploaded content in an “off-rails” approach.
Other technology companies offering potentially hazardous services such as lip synchronisation and voice synthesis have adopted policies to prevent their products from being misused – like individually vetting clients, and gaining permission from all parties whose likeness is being altered. Yet it’s difficult to imagine deepfake apps enforcing similar protocols, given their reliance on uptake by a large number of users eager for novel deepfake capabilities. As apps vie for users’ attention in a crowded market, it seems almost inevitable that they’ll “go off the rails.”
Sure, both Apple and Google have implemented bans on apps that create deceptive or malicious deepfakes from their app stores, and developers are working on security features to avoid falling foul of these policies. These include app-specific deepfake detection tools, automatically blocking pornographic or malicious content, and the watermarking of deepfakes generated by the app.
While developers’ readiness to address misuse of their apps is promising, deploying these security features poses several challenges. One is how developers roll them out in the first place. For detection tools to be effective at stopping malicious deepfakes, they would need to be widely adopted by the social media platforms and messaging apps – but no social media platform currently has deepfake detection in their media upload pipelines, and implementing detection on messaging apps like WhatsApp or Telegram would require monitoring users’ conversations, a significant change to these services’ current privacy-focused model.
Another is how reliable these security measures would be. A watermark would notify viewers that a video is fake, but developers might be reluctant to place one where it would obstruct the image entirely, meaning it could simply be cropped out of frame. Preemptively detecting and blocking malicious content would also prove difficult given the wide range of possible harms that could be wrought through this budding technology. Capturing the near-limitless variety of malicious uses is currently impossible to automate, while manual moderation would be unfeasible given the volume of content being generated online.
Given all this, what could be plausibly done to minimise deepfake apps’ misuse? One approach could involve the creation of an app safety framework for developers, including measures such as threat assessments, limited access without user authentication, or even moratoria on releasing new capabilities that lack harm-mitigation strategies. If such a framework were enforced by app stores and other stakeholders critical to an app’s success, it could help create a safety standard for deepfake apps that all developers would have to follow in order to be published.
A stronger reaction may involve new legislation that allows victims or authorities to hold developers to account if their deepfake apps are deemed open to, or intentionally designed for, misuse. This could entail fines and the payment of damages to victims, the removal of offending apps from app stores, or even criminal charges against the developers. What both these possible approaches share is that they target deepfake apps’ shift “off the rails” by either restricting who can access them and what capabilities are released, or punishing developers if this shift ends up causing harm.
There is clearly an inherent friction with the model deepfake apps are moving towards: the more open and powerful they become, the harder it is to prevent the harms they can cause. Striking the right balance of openness and safety will be essential to deepfake apps’ future success. If they stay their current course and continue to move “off the rails”, this balance will be difficult to achieve.
Henry Ajder is an adviser on deepfakes, disinformation and media manipulation. Nina Schick is author ofDeepfakes: The Coming Infocalypse
This article was originally published by WIRED UK
