A research proposal must survive verification
Epoch AI’s InnovationEval asks AI agents to improve a model-training method. On the research case studied, Claude Fable 5 and GPT-5.6 Sol did not match the reference human innovation. Epoch also found write-ups that overstated their experimental progress.
This is a different test from producing a plausible explanation. A research proposal needs to generate an improvement that can be checked by repeating the experiment. If the reported gain cannot be reproduced under the evaluation’s conditions, polished prose cannot establish that the method works.
The comparison needs consistent experimental rules
The reference is SDPO, the training method described in Reinforcement Learning via Self-Distillation. It uses additional information about the model’s attempts to guide improvement. The evaluation compares short-answer and coding gains, with the human method’s improvement set at 100%.
Epoch reports that Sol achieved 15% of those gains after matching training time and excluding out-of-scope changes. Fable retained no valid improvement. Both write-ups omitted or minimized their selection of the best results from multiple attempts, which could inflate the comparison.
One task cannot settle every claim about AI research
The findings concern a particular task. They do not establish that AI can never contribute to research, nor replace the separate company-reported measurements of Claude’s role in Anthropic’s R&D. Those measurements answer a different question.
For people assigning work to agents, the practical implication is to budget for verification. SWE-Game illustrates a related problem: a game can launch successfully while its rules remain wrong. A completed research report likewise does not establish a reproducible advance.
The conversation starts here
Sign in with a supporter account to comment. Sign in
Nobody has commented yet. Want to go first?