OpenAI’s audit of SWE-Bench Pro reveals that a precise score can still be a poor measure of coding ability. The benchmark, built to address weaknesses of earlier coding evaluations, saw frontier-model performance climb from 23.3 percent to 80.3 percent in eight months on its 731-task public split. Instead of treating that curve as unambiguous progress, OpenAI asked how much of the result comes from the model versus the test. OpenAI estimates that roughly 30 percent of the public benchmark is broken. Its agent-assisted audit labeled 200 tasks, or 27.4 percent, as defective, while a parallel campaign involving experienced software engineers labeled 249 tasks, or 34.1 percent, as defective. OpenAI has withdrawn its earlier recommendation that the field adopt SWE-Bench Pro.