Releasing AISI’s Engineering Playbook

The AI Security Institute has released the AISI Engineering Playbook, providing an open resource that captures the methods and practices developed over two years for evaluating frontier AI systems. This release builds on the momentum of Inspect AI, a toolkit open-sourced in May 2024 to evaluate frontier model capabilities. According to the AISI Blog, evaluating frontier AI at scale requires complex supporting infrastructure that is often invisible, rarely documented, and built from scratch by individual teams. By open-sourcing its methods and infrastructure alongside Inspect, AISI hopes to help researchers and organizations establish rigorous evaluation capabilities without starting from zero. The Engineering Playbook breaks the wider infrastructure of evaluation into five layers: Evaluate, Isolate, Connect, Run, and Scale. It offers technical overviews, deep dives into each capability section, and links to open-source code. The companion project Inspect Evals provides a collection of over 200 pre-built evaluations covering coding, agentic tasks, reasoning, knowledge, behavior, and multi-modal understanding. Since its launch, the Inspect community has grown to include over 240 contributors to inspect_ai. AISI states that the Playbook represents a snapshot of its current evaluation practices and will be updated as methods and infrastructure evolve. The institute believes that sharing this technical stack is essential for maintaining quality evaluations as frontier AI grows more capable and complex.