Why I joined AISI by Geoffrey Irving

Note to readers: we changed our name to the AI Security Institute on 14 February 2025. Read more here.

I’ve been working on AI and machine learning for 10 years, including leading the Reflection Team at OpenAI and the Scalable Alignment Team at DeepMind. But this year, for the first time, I took a job in the government and joined the UK AI Safety Institute. People often ask me why I did this (often because they are considering a similar leap).

I continue to think direct technical work on AI and AGI safety is critically important: we have a ton of hard sociotechnical challenges, but I think good solutions will be available given sufficient time.

In contrast, at AISI I am doing technical work motivated by policy, with two broad goals:

First, I worry that progress on safety is too slow relative to the rate of advancement in AI. This is true both for conceptual and theoretical safety work and for empirical methods. I am hopeful about the in-principle effectiveness of many safety approaches given sufficient time, but we do need that sufficient time. Even restricting to known unknowns: all existing technical safety approaches have key conceptual or empirical milestones to hit before we can confidently rely on them. Many of these problems have been known for years and progress against them has been slow.

There is thus immense value in lifting risks and uncertainties into the open, so that society and governments can make informed decisions. Most employees and leaders at AI labs mean well, but market forces mean that there is constant pressure to prioritise speed over caution. I believe governments have a key role in enabling coordination towards safer actions. Race dynamics can exacerbate risks, but are not inevitable.