People Are Using Sora 2 to Make Disturbing Videos With AI-Generated Kids
The Dark Side of AI-Generated Content: How Sora 2 is Being Used to Create Disturbing Videos
The rise of AI-generated content has opened up new possibilities for creators, but it has also raised concerns about the potential for misuse. One of the most disturbing examples of this is the use of Sora 2, a video generator developed by OpenAI, to create videos that feature AI-generated children in compromising situations.
The Vibro Rose: A Fake Toy Commercial that Went Too Far
On October 7, a TikTok account named @fujitiva48 posted a video that appeared to be a TV commercial parody. The video showed a photorealistic young girl holding a toy, a pink, sparkling pen with a bumblebee design, that looked and sounded suspiciously like a sex toy. The video was intended to be humorous, but it quickly became clear that it was not funny at all.
The Response was Immediate and Overwhelming
The video elicited a strong reaction from viewers, with many expressing outrage and disgust. "Hey so this isn’t funny," wrote one person. "Whoever made this should be investigated." The video was quickly removed from TikTok, but not before it had gone viral and sparked a wider conversation about the potential for AI-generated content to be used for nefarious purposes.
The Laws on AI-Generated Fetish Content Remain Blurry
The laws on AI-generated fetish content involving minors remain blurry, and it is unclear what constitutes a violation. New data from the Internet Watch Foundation in the UK notes that reports of AI-generated child sexual abuse material have doubled in the span of one year, with 56% of this content falling into Category A, the UK’s most serious category involving penetrative sexual activity, sexual activity with an animal, or sadism.
OpenAI's Measures to Prevent the Misuse of Sora 2
OpenAI has implemented measures to prevent the misuse of Sora 2, including a feature that prevents young people from having their faces plastered onto pornographic deepfakes. The app's feature, which was previously called Cameo but is now temporarily renamed, works on a consent basis and can be revoked at any time. OpenAI has also banned CSAM and child endangerment on its platforms and reports any child sexual abuse material and child endangerment to the National Center for Missing and Exploited Children.
But What About the Rose Toys?
While OpenAI's measures are a step in the right direction, they do not seem to be enough to prevent the creation of videos like the rose toys. The videos are conspicuous enough in their signposting for TikTok commenters, YouTubers, and content creators to invite serious discourse over their dangers, yet creators seem to be able to make them by circumventing OpenAI's guardrails.
The Contextual Nuance of CSAM
The contextual nuance of CSAM is a major challenge for AI companies like OpenAI. The videos of the young overweight boys, for example, are not explicitly pornographic, but their intention becomes clear when viewing the video's comments section, where several accounts had shared requests for people to add them on Telegram, which law enforcement has claimed is a hub for pedophile networks.
The Need for Nuance in Moderation Practices
Mike Stabile, public policy director at the nonprofit Free Speech Coalition, believes that OpenAI and similar companies must implement a similar level of nuance in their moderation practices to fully ensure CSAM and fetish content featuring children doesn’t make it onto the platform. This could involve banning or limiting certain words that might be associated with fetish content and improving moderation teams with more diversity and training.
The Struggle with Platforms Like Facebook
Stabile notes that platforms like Facebook already struggle with differentiating between a parent sharing a picture of their kid playing in a pool or the bath versus somebody who's sharing something that's meant to be child sex abuse material. "Anytime you're dealing with kink or fetish, there will be things that people who are not familiar are going to miss," he says.
OpenAI's Response
Following WIRED's request for comment, OpenAI said it banned several accounts that were creating videos like the vibrating rose toys. Niko Felix, spokesperson from OpenAI, told WIRED: "OpenAI strictly prohibits any use of our models to create or distribute content that exploits or harms children. We design our systems to refuse these requests, look for attempts to get around our policies, and take action when violations occur."
TikTok's Response
A TikTok spokesperson said: "We have removed videos and banned accounts that uploaded content created on other AI platforms which violated TikTok's strict minor safety policies." However, many of the videos and accounts that WIRED flagged remain online at the time of reporting.
The Need for Products and Platforms to be Safe by Design
The Internet Watch Foundation urges platforms to consider these safeguards in their initial design and put them at the forefront of their priorities—preventing harmful content from being made to begin with. "We want to see products and platforms which are safe by design, and encourage AI companies to do as much as they can to make sure their products can not be abused to create child sexual abuse imagery," Smith tells WIRED.
Conclusion
The use of Sora 2 to create videos that feature AI-generated children in compromising situations is a disturbing example of the potential for AI-generated content to be used for nefarious purposes. While OpenAI has implemented measures to prevent the misuse of Sora 2, they do not seem to be enough to prevent the creation of videos like the rose toys. The contextual nuance of CSAM is a major challenge for AI companies, and the need for nuance in moderation practices is clear. Platforms like TikTok and Facebook already struggle with differentiating between legitimate and illegitimate content, and the struggle will only continue as AI-generated content becomes more prevalent.
Source: https://www.wired.com/story/people-are-using-sora-2-to-make-child-fetish-content/
