Disrupting AI-enabled “false front” operations

Over the past two and a half years, we’ve reported on many of the ways in which threat actors tried to use our models to conduct cyber attacks, covert influence operations (IO), scams, and other violations of our Usage Policies. We publish these reports to inform regulators, our industry peers, and wider society of how we see threat actors trying to leverage AI, and to shine a light on the vulnerabilities they try to exploit.

We recently banned two IO—one from Russia and one from Iran—that used our models in combination with more traditional techniques and technologies to support sophisticated “false front” entities. They used those entities to launder geopolitical, conflict-related messaging into their target audiences. The Iranian operation included a stable of seven “journalist” personas which it used to pitch long-form articles to small and medium online outlets around the world; the Russian operation appears to have co-opted unwitting people in Latin America to run a “think tank” on the ground.

Both operations were multi-dimensional, trying a range of different tactics to achieve their goals. As well as long-form articles, the Iranian operation generated batches of social media comments, generally on topics related to the US-Iran war. As well as controlling the “think tank,” the Russian operation created fake “leaked” documents and audio scripts, some of which we identified being spread online. Both operations also made heavy use of AI to draft internal reports (the Russian operation did this more than anything else); in both cases, the actors used questionable or outright deceitful methodologies to exaggerate the operators’ effectiveness.

What is most striking about these operations is that they closely resembled complex influence operations of the pre-AI age, but used AI to make some of the workflows easier. The journalist personas run by the Iranian operation bore a family resemblance to the fake journalist “ Alice ⁠ (opens in a new window) Donovan ⁠ (opens in a new window) ,” a front for Russian military intelligence whose articles were published by a range of Western outlets in 2016-17. In 2020, individuals associated with past activity by the Russian Internet Research Agency ran a fake “news” outlet called “ PeaceData ⁠ (opens in a new window) ,” which co-opted unwitting journalists around the world into writing for it.

These operations were also unusual for their potential reach. Using the IO Breakout Scale ⁠ (opens in a new window) , which rates IO on a scale of 1 (lowest) to 6 (highest), we would assess the Russia-origin operation as belonging in Category 5 . This is the first Category 5 operation we’ve disrupted since we began our reporting. The Iran-origin operation reached Category 4 . Both managed to land their content (not all of which was generated from our models) in mainstream media outlets, rather than simply posting it on social media. This is consistent with a pattern we’ve observed across the 30 covert influence operations we’ve exposed in the last two and a half years: the operations which try to land their content in real media outlets, rather than relying on fake social media distribution, tend to have the highest potential reach and impact.

Matrix of the 30 IO we’ve exposed since early 2024, assessed by primary distribution method (social media, operation-run website, external publications) and score on the Breakout Scale. Operations which used more than one distribution method, such as running a website and promoting it on social media, are classified by the method which appeared to constitute the core of the operation.