San Francisco tells Apple and Google to remove 13 fake-nude apps as an EU ban approaches

California’s notice starts a 30-day legal clock

San Francisco city attorney David Chiu sent cease-and-desist letters to Apple and Google on Thursday, July 16. The letters, reviewed by WIRED, cover 13 face-swapping products—eight on Apple’s App Store and five on Google Play. Investigators say the apps could place a real person’s face on a nude body or create a fabricated sexual scene without that person’s consent.

This is not yet a court judgment or a worldwide removal order. Chiu has put the companies on notice and can sue if they do not act. Both have already begun responding.

The legal leverage comes from California’s AB-621, which took effect in 2026. It reaches beyond the operator that creates or distributes a sexual deepfake. A service provider can be presumed to facilitate the business if it receives sufficient evidence, keeps supplying an essential service and fails to cut that relationship within 30 days. Public prosecutors can seek injunctions and civil penalties of $25,000 per violation, rising to $50,000 for malicious conduct.

App distribution and in-app payments sit squarely inside that argument. Apple and Google hosted the software, processed purchases and kept a share of the revenue. Chiu’s office says that makes them more than passive shop windows.

The stores had rules—the apps still passed review

Google told WIRED it had removed all five Android apps in the letters. The company says it has deleted hundreds of products with nudification features and restricted related search terms. Apple confirmed that it had removed three apps and was terminating the developers’ accounts. Four other developers must fix policy violations or face removal; Apple’s public response did not explain the status of the eighth iOS app.

Neither store needed a new content rule. Google Play’s policy explicitly rejects apps that claim to undress people, see through clothing or create non-consensual sexual deepfakes. Apple’s review guidelines prohibit pornographic material, exploitation and apps primarily used to objectify or bully real people.

The review problem begins when the harmful capability does not appear in the listing. An app can present itself as a playful face editor, photo animator or virtual-makeup tool, then reveal a different function after installation or payment. Checking the icon, promotional screenshots and a developer-controlled demo cannot show everything the software will accept.

Publishing the 13 names would also create a directory and quickly become obsolete. Developers can change an icon, relaunch under a new account or move the same function behind a different label. Effective review requires testers to attempt prohibited uses before approval and again after significant updates.

Researchers tested 155 face-swap apps; 109 failed every check

A May Cornell and Georgetown preprint provides the strongest independent measure. Its authors found 420 apps by searching “face swap” across both stores. They could apply a common test to 155: four research-generated image pairs spanning two skin tones and two sexes. For each pair, they attempted to place the face of a clothed fictional person onto a fictional nude body.

The test did not include products openly marketed as undressing tools. Even so, 109 of the 155 apps—70%—completed all four sexual face swaps. Sixteen blocked some image pairs but allowed others. Only 30 refused all four. The failure rate reached 80% on iOS and 58.6% on Android.

Partial blocking introduced another concern. In half of the observed failure patterns, the filter activated only for images depicting people with darker skin. A safety system that accepts one near-identical input and rejects another may be responding to lighting, skin tone or gaps in its training data rather than applying a dependable rule.

The sample has boundaries. A single keyword cannot discover every app; 265 products were excluded because they lacked the exact two-image feature, failed to run or fell outside the procedure; and the researchers used generated test material. The figures describe 155 products at one point in time, not today’s complete market. Yet none advertised itself as a nudification app, while 109 accepted every attempted explicit swap.

The authors recommend adversarial testing before publication, nudity filters, deterrence messages and fresh checks after major updates. INSERT FUTURE’s chart reproduces the numerical findings without using the paper’s sensitive test imagery.

EU distribution rules change on December 2

San Francisco is applying one state’s law to two US companies. Europe has chosen a market-wide rule. The European Parliament approved an amendment to the AI Act in June, and the Council gave final approval on June 29. It prohibits AI systems intended to generate child sexual abuse material or non-consensual depictions of an identifiable person’s intimate parts and sexual activity.

General-purpose editors may remain available if they include adequate technical measures preventing those outputs. A legitimate photo editor is not banned merely because it can swap a face or fill part of an image. It must stop those capabilities becoming an on-demand fake-nude service. Providers have until December 2, 2026 to comply.

That deadline covers Spain and the rest of the EU. Renaming a function will not be enough if the same request still works. The change follows the regulatory direction covered in INSERT FUTURE’s analysis of converging AI rules: governments increasingly expect the companies distributing an AI system to answer for how people can use it.

The EU ban will not erase downloadable models, private bots or services hosted beyond app stores. It does, however, put pressure on the two gateways most phone owners trust. Delisting removes reach, payment processing and the implied reassurance of store review.

Removing an app does not remove an image

Stopping new downloads cannot recover photographs already uploaded to an app’s servers or delete copies someone has saved and shared. Spain’s data protection authority warns that the loss of control begins when somebody uploads another person’s image: invisible retention, copies and additional processing can happen even when the final edit is never published.

People in Spain facing the online spread of non-consensual sexual imagery can use the AEPD’s Priority Channel to request urgent removal. The affected person or somebody aware of the case can file. The authority asks for the relevant URLs and circumstances and may alert prosecutors where it finds evidence of a crime. INCIBE’s free 017 service can guide victims and families; threats, extortion, harassment or cases involving children should also be reported to law enforcement.

San Francisco has named 13 apps and prompted the first removals. The next test is whether Apple, Google and European regulators can look beyond an app’s name and repeat the researchers’ practical checks after every important update. We will track that result in INSERT FUTURE’s AI channel as December approaches.