Separating signal from noise in coding evaluations

OpenAI has released an analysis of SWE-Bench Pro, a popular coding benchmark designed to evaluate agentic coding capabilities on longer horizons and realistic tasks. Following a previous investigation into SWE-Bench Verified, OpenAI audited SWE-Bench Pro using a datapoint analysis pipeline, human-supervised agent reviews, and a human annotation campaign involving experienced software engineers. The analysis revealed that approximately 30 percent of SWE-bench Pro tasks are broken, with the automated pipeline flagging 27.4 percent and human reviewers identifying 34.1 percent. The identified issues include overly strict tests that enforce specific implementation details, underspecified prompts that omit hidden requirements, low-coverage tests that allow incomplete fixes to pass, and misleading prompts that point models toward the wrong behavior. OpenAI noted that issues arise because tasks derived from open-source repositories were originally created for human collaboration rather than clean, isolated AI evaluations. Because accurate measurements are crucial for sound deployment and safety decisions under OpenAI's Preparedness Framework, the organization has retracted its earlier recommendation to adopt SWE-Bench Pro and advises model developers to carefully examine their results.