Our approach to tackling AI-generated child sexual abuse material

Today, AISI and child safety organisation Thorn are publishing a new safety protocol to help prevent the creation and spread of AI‑generated child sexual abuse material (CSAM). The protocol sets out practical steps that AI developers, model hosting services and others in the AI ecosystem can take to reduce the risk that their systems are misused to generate CSAM.

This comes alongside new legislation which AISI has helped to shape that will, under strict conditions, allow authorised organisations to test whether AI models can be used to generate AI‑generated CSAM, so that they can strengthen safeguards and reduce the risk of misuse. Together, the protocol and the legislative change form an important part of the UK’s wider approach to tackling the growing threat of AI‑generated CSAM.

Possessing and generating child sexual abuse material is already illegal under UK law, regardless of how it is created. That includes material made or altered using AI, image-editing software or any other editing tools, whether it depicts real children, uses their likenesses, or fabricates fake children entirely synthetically. But rapid advances in AI image and video tools are changing the scale and nature of the problem.

AISI will work alongside the Home Office to consult with experts from industry, child safety organisations, law enforcement and academia to design the conditions and operational safeguards that will apply to this newly permitted testing. These safeguards will help prevent sensitive content from leaking, set clear parameters for testing activity, and protect the wellbeing of researchers.

AI is increasingly being used to generate realistic, synthetic child sexual abuse images which are them disseminated. A recent report from the Internet Watch Foundation (IWF) found 3,512 AI‑generated CSAM images on a single dark‑web forum in one month. This material harms real children, whose likenesses are depicted in abuse scenarios, and adds to the burden of law enforcement who, on top of protecting children and bringing offenders to justice, must process larger volumes of content and identify where images depict real children in real danger.

Because creating or possessing CSAM is currently illegal in all contexts, developers and children protection organisations cannot test AI models directly to see whether they can be misused to generate this material. As a result, most companies have had to rely on indirect and imperfect methods.

The measures introduced in November will change this . They will enable the Government to authorise a small number of organisations – such as AI developers and third sector organisations – to test whether AI models can be used to generate CSAM under strict conditions, helping to limit the production of AI-generated CSAM in the first place.