Import AI’s newsletter highlights DiG‑bench, a new benchmark comprising 70 text‑based games created by experts from institutions such as Oxford, Princeton, MIT, and Swiss AI Lab. Each game is a self‑contained miniature world with hidden rules and objectives, designed to assess how well AI systems can infer unwritten mechanics through exploration rather than being explicitly told. The benchmark is divided into seven tiers of increasing difficulty; 21 games are publicly released while the rest remain private to prevent training bias. Current frontier models—Opus 5, Fable 5, and GPT‑5.5—have achieved the highest overall performance, with Opus 5 and Fable 5 the only models to beat any Tier 7 tasks, achieving a 0.2 success rate. In Tier 6, Opus 5, GPT‑5.5, and Kimi K3 can beat some tasks when given a harness such as Claude Code. Tier 4 saw GLM‑5.2 and Gemini 3.1 Pro beat some levels. The article notes that these models still lag far behind humans, who achieve 100% success on Tier 7, and predicts human parity on DiG‑bench by mid‑2027. The piece also discusses broader concerns about superhuman invention and individual empowerment, citing Meta’s perspective on how AI could shift power balances. This discussion underscores the importance of measuring AI creativity and discovery to anticipate future societal impacts.
Import AI 469: Science AI; RSI simulator; and Zuck's technological pessimism
Import AI reports on DiG-bench, a benchmark of 70 text‑based games designed to test AI’s ability to infer hidden rules through exploration. The benchmark, developed by researchers from Oxford, Princeton, MIT, and others, includes seven difficulty tiers and measures how well models can discover game mechanics. The article notes that current frontier models such as Opus 5, Fable 5, and GPT‑5.5 achieve limited success, with only Opus 5 and Fable 5 beating Tier 7 tasks at a 0.2 success rate. It also discusses broader concerns about superhuman invention and individual empowerment, citing Meta’s perspective on AI’s societal impact.
Why it matters
Understanding AI’s ability to autonomously discover hidden rules informs progress toward creative and adaptive systems. The benchmark reveals current limits and guides future research on bridging the gap to human-level discovery.