Paper
Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
‡ Project Lead † Corresponding author: chennuo26@gmail.com
Contact: Bowen Liu · bliu690@connect.hkust-gz.edu.cn
- Tencent Hy Team
- The Hong Kong University of Science and Technology
- The Hong Kong University of Science and Technology (Guangzhou)
- Tsinghua University
- Peking University
Manuscript draft · 2026. The arXiv record is forthcoming.
Abstract
Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. We evaluate 12 coding agents, each pairing a model with a harness. Of their 325 correct outcomes, 161 (49.5%) stay below 1.25x speedup. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.