A shared playbook for trustworthy third party evaluations

OpenAI has shared guidance and lessons learned regarding independent third-party evaluations for frontier AI systems. Because modern frontier models can utilize tools, maintain information across multiple steps, and act within broader workflows, their performance depends heavily on the surrounding setup, or harness. According to the guidance, useful evaluation reports must explicitly state the claim being tested and provide evidence of the result's validity. Claims typically fall into three categories: capability elicitation, safeguard performance, and controlled comparison. Evaluators must also check for validity issues such as reward hacking, refusals, data contamination, broken problems, and sandbagging. The document notes that the harness and allocated budget significantly impact observed performance. For instance, choices regarding task-relevant context preservation, test-time compute, and token budgets can change measured capability, meaning that scores reflect performance under specific conditions rather than fixed capability ceilings. Consequently, avoidable under-elicitation represents a measurement failure. OpenAI hopes these recommendations will help inform emerging industry standards for validly assessing frontier AI models.