Epoch and METR’s MirrorCode benchmark evaluates how well AI systems can reimplement software programs solely through command‑line interaction, without source code or web access. The benchmark includes 25 target programs, such as pkl, gotree, and qsv_select, and measures success across six programming languages. Opus 4.7 solved a 14‑hour task for $251 in inference cost, a job that would take humans 2–17 weeks, and successfully reimplemented programs like gotree and pkl. However, 8 of the 25 targets never reached a 100% score, with the Python linter ruff and the mathematics package giac_subset proving particularly challenging. In robotics, Anthropic’s Claude Opus 4.1 initially failed to complete quadruped tasks, but Opus 4.7 autonomously completed all but one task in 9 minutes and 35 seconds, outperforming humans who took 181 minutes. Sunday’s ACT‑2 model demonstrates that scaling pre‑training and minimal in‑house data can yield high reliability in household robotics, achieving a 99.1% success rate in garment folding across nine types. These developments suggest that as large language models improve, they may unlock broader capabilities in software development and robotics, enabling more autonomous and general‑purpose AI systems.
Import AI 466: The bitter lesson for robotics, AIs complete week-long programming tasks; and OpenAI's accidental AI hacker
Epoch and METR released MirrorCode, a benchmark that tests AI systems’ ability to reimplement long‑horizon software programs from command‑line interaction alone. Opus 4.7 and GPT‑5.5 achieved high success rates on several target programs, with Opus 4.7 completing a 14‑hour task for $251 in inference cost, a job that would take humans 2–17 weeks. Anthropic’s Claude Opus 4.7 also demonstrated rapid progress in autonomous quadruped robot tasks, completing most tasks in under 10 minutes compared to 181 minutes for humans. Sunday’s ACT‑2 model shows that scaling pre‑training and minimal in‑house data can yield high reliability in household robotics, achieving 99.1% success in garment folding.
Why it matters
These results illustrate that large language models are rapidly gaining the ability to understand and replicate complex software and robotic behaviors, suggesting a path toward more autonomous, general‑purpose AI systems that can operate in diverse real‑world environments.