What happened?
Hugging Face organized a hackathon called "ICML 2026 Open Reproductions" from July 15 to August 2, 2026. 1,221 community members used coding agents such as Claude Code, Codex, Cursor, Pi, and OpenResearch's orx tool to attempt claim-by-claim reproduction of papers accepted at the International Conference on Machine Learning (ICML) 2026.
Over the 19-day event, participants published 6,816 Trackio logbooks and put 2,226 of the 6,341 accepted papers—roughly a third of the conference—to the test. Each logbook was published as a static Hugging Face Space page containing the code that was run, the outputs produced, and, optionally, the agent's full transcript. Participants were given $20 in cloud compute credits to run experiments on Hugging Face Jobs; a total of 2,962 cloud jobs were launched.
Why does it matter?
ICML 2026 received 23,918 submissions, of which 6,352 were accepted—roughly double the previous year's figure. Conference organizers partly attribute this surge to AI agents, which let researchers run and write up experiments faster. Because the number of volunteer reviewers did not grow at the same pace, one reviewer admitted to not carefully checking the evidence—yet the paper still received a spotlight award, a fact that surfaced during the hackathon.
Every logbook was re-read by an automated "Logbook Judge" system running on an open-weight model called GLM-5.2, which marked each claim as verified, refuted, toy-scale (supported only by limited data), or inconclusive. The system was designed to treat each logbook's self-assessment as unreliable. In total, 35,908 claims were reviewed this way, and 274 full agent transcripts were released as a dataset.
How did the results break down?
Verification results aggregated at the paper level can be summarized as follows:
| Category | Number of papers | Share |
|---|---|---|
| At least one claim verified | 1,103 | 51% |
| All claims verified (full reproduction) | 266 | - |
| Partially verified, none refuted | 632 | - |
| At least one claim refuted/disputed | 496 | 23% |
| All claims refuted | 49 | - |
| Different teams reached conflicting results | 242 | - |
| Only toy-scale evidence | 502 | - |
| Inconclusive | 280 | - |
In total, 3,978 individual claims were verified through actual experiments. The paper "Flat Minima and Generalization: Insights from Stochastic Convex Optimization" was reproduced by 20 different teams, 12 of which verified all claims. For "A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness," 14 of 17 logbooks confirmed all claims.
Which papers were refuted?
35 participants formally reported refuting a claim; the Hugging Face team confirmed all of these refutations by re-examining the paper, the logbook, and the mathematical derivation. Three notable examples:
- "Towards Optimal Robustness in Learning-Augmented Paging": The paper claimed its algorithm achieved H_k + O(1) robustness. A participant's logbook found that the additional term actually grew at a rate of 0.38 ln k and pinpointed exactly where the proof broke down; Hugging Face's extended test up to k=1,024 confirmed this growth with roughly nine-sigma confidence.
- "Attention's forward pass and Frank-Wolfe": The paper proved that token particles collapse to the origin under certain conditions. Three independent teams found counterexamples at 224, roughly 3,800, and 6,416 steps respectively—explaining why other teams had "verified" the claim, since their limited-horizon tests stopped before reaching those steps. The authors confirmed the finding the same day and are working on a correction.
- "Self-Distillation Enables Continual Learning": While the paper's theoretical section was built on reverse Kullback-Leibler divergence, the published code's default setting computed forward KL instead. The logbook that reported the refutation could not reproduce the paper's headline +4-point result even using the authors' own code and data. The authors have since uploaded an updated version to arXiv.
What's next?
Hugging Face has publicly released the dataset of 35,908 claims collected during the hackathon along with all the logbooks. The team says the role of automated verification agents in the scientific review process will keep growing, and that similar mass-reproduction efforts could be repeated for future conferences. The company has not announced a timeline for which conferences will undergo similar scrutiny next.