xAI is suing Minnesota's Attorney General over a recently passed law aimed at regulating "nudification" apps, which the company claims violates the First Amendment. The law, signed in May, targets software that allows users to create non-consensual nude images and carries penalties for violations. The law is seen as overbroad by xAI, which argues it also criminalizes consensual imagery and restricts freedom of expression. Following significant public backlash and scrutiny regarding its app, Grok, which generated millions of sexually explicit deepfakes, the company seeks to invalidate the law before it takes effect in August.

Notifications

Hamburger Navigation Button

Navigation Drawer

The VergeThe Verge logo.

closeClose

LightSystemDark

Comments Drawer

Notifications

Comments

Loading comments

Getting the conversation ready...

Policy
AI
Speech

xAI’s last-minute scramble to stop Minnesota’s anti-nudification app law

Grok, is this statute constitutional?

bySarah Jeong

Jul 29, 2026, 5:06 PM EDT

Link
Share
Gift

STK022_ELON_MUSK_CVIRGINIA4_I

Image: Cath Virginia / The Verge, Getty Images

Part Of\ \ The latest on Grok’s gross AI deepfakes problem\ \ see all updates

Sarah Jeongis a reporter who writes about law, technology, speech, and democracy. A journalist trained as a lawyer, she has been writing about tech for 10 years.

xAI is suing Minnesota Attorney General Keith Ellison over a law passed back in May that broadly targets “nudification” apps, claiming that the statute’s punitive provisions leave the company with “no practical choice but to restrict Grok Imagine’s image-editing features in various ways.” The law, the company argues, violates the First Amendment.

Back in January, Grok flooded the internet with millions of sexually explicit deepfakes, including images of minors. In another era, a content moderation disaster of this magnitude would have had the CEO flipping the kill switch on the product within hours. But in Elon Musk’s brave new world, the chatbot was left to pump out pornographic images for days. “Grok now hitting #1 on the App Store in one country after another!” he posted in the midst of the deepfake epidemic.

Add Verge on Google

Add Verge as a preferred source to see more of our reporting on Google.

Add us on Google

Something had gone wrong. Was the law simply not up to the task of curbing an explosion of nonconsensual nudes? Governments around the world were asking fairly similar questions. The EU launched an investigation, as did the United Kingdom. There was backlash from French, Indian, and Malaysian government officials. And while the American federal government was noticeably quieter on the issue, blue states like California and Minnesota took a harder line.

In Minnesota, the state legislature passed a law targeting “nudification” technology, prohibiting site owners from allowing people to “access, download, or use” such software. (The law makes a carve-out for tech that “requires the technical skill of a user to nudify an image or video” — so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.

The law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It’s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It’s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.

The complaint is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with “humorous” or “artistic” merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.

A real excerpt from page 20 of the lawsuit.

“Liability also attaches … even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,” the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users — “images and videos that users might manage to create in violation of Grok’s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.”

“xAI takes violations of its terms of service seriously,” the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)

xAI’s lawyers might have a point, but their lawsuit reads like a missive from another planet. They claim, for instance, that existing Minnesota and federal law (like the TAKE IT DOWN Act) already prevents the nonconsensual dissemination of nude deepfakes. These statutes, the lawyers argue, are written in a constitutional way — the nudification law, on the other hand, is not. And as with any laws in America that target speech based on its content, these statutes must be written in as unrestrictive a way as possible while still addressing a “compelling” government interest.

But that’s of course the crux of the problem with the Grok deepfake-ocalypse. If existing law was up to the task, the entire month of January 2026 wouldn’t have been awash with nonconsensual porn. When the Center for Countering Digital Hate analyzed an 11-day period from December 29th to January 8th, it found that Grok had generated around 3 million sexualized images. Around 23,000 of those images depicted children. “That is a shocking rate of one sexualized image of a child every 41 seconds,” the report read.

A lot of those images might not even be actually illegal. For instance, it’s not clear that a suggestive deepfake of a minor in a bikini is CSAM, although one expert previously told The Verge that other images Grok generated — like ones showing minors’ faces covered in semen — probably did violate existing law.

Either way, the mere fact that a multibillion-dollar company allowed Grok to run unchecked for that long would suggest that the law was not motivating the relevant parties to behave in the right way. Maybe Minnesota’s HF 1606 is a bad law. But what does that mean when the “good” laws aren’t even working?

Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.

Sarah Jeong
AI
Policy
Speech
xAI

More in: The latest on Grok’s gross AI deepfakes problem

xAI sues a man for using Grok to generate CSAM ‘deepfakes’

Emma RothJul 15

Grok’s sexual deepfakes almost got it banned from Apple’s App Store. Almost.

Robert HartApr 15

Teens sue Elon Musk’s xAI over Grok’s AI-generated CSAM

Emma RothMar 16

Most Popular

The Verge Daily

A free daily digest of the news that matters most.

Email (required)

By submitting your email, you agree to our Terms and Privacy Notice. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Advertiser Content From\ \ Sponsor Logo \ \ This is the title for the native ad\ \ Sponsor thumbnail