# PaperBench > Benchmark where AI agents must replicate 20 ICML 2024 papers end-to-end; best AI reached 21% vs. expert human baseline - Authors: Hunter - Published: 2025-04-11 - Source: Bevaya Labs, --- # PaperBench: Evaluating AI's Ability to Replicate AI Research **Paper**: [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://arxiv.org/abs/2504.01848) **Presenter**: Hunter **Note Taker**: Hunter **Date**: April 11, 2025 --- ## Overview PaperBench is a benchmark that tests whether AI agents can replicate cutting-edge ML research end-to-end. Agents are given a paper and must reproduce it from scratch: understanding contributions, writing code, and running experiments. The benchmark covers 20 ICML 2024 papers decomposed into 8,316 individually gradable sub-tasks, with rubrics co-developed with original authors. The best-performing agent (Claude 3.5 Sonnet with open-source scaffolding) achieved 21.0%; top ML PhD students still outperform current AI systems on evaluated subsets. --- ## Novelty - Benchmark requiring full replication pipeline: comprehension, implementation, and experiment execution, not just code generation or QA. - 8,316 hierarchically decomposed sub-tasks with author-validated rubrics for fine-grained evaluation. - Automated LLM-based judge benchmarked separately to assess grading reliability. - Human baseline (ML PhDs) established on a subset to anchor the difficulty scale. - Open-source code and scaffolding released alongside the benchmark. --- ## Learnings - Current AI agents are far from reliably replicating research: even the best scores only 21%, with most agents in the single digits. - Task decomposition is important: coarse pass/fail on a full paper hides meaningful partial progress visible only at the sub-task level. - LLM-based judges can be reliable graders when calibrated, but their reliability must itself be measured. PaperBench reports judge accuracy separately. - Human expert performance is still substantially above AI, establishing a clear ceiling to work toward. --- ## Notes for Application - The benchmark framing (hierarchical rubrics, author-validated) is a useful pattern for internal evaluation of model-generated outputs: break complex tasks into sub-tasks and score each independently. - Automated LLM judges work for grading open-ended generation, but their calibration should be validated against human labels for high-stakes decisions. - The 21% ceiling underscores that current agents are unreliable for autonomous research replication; human review remains essential for complex multi-step tasks.