Correct programs often gain little.
161 of 325 correct outcomes stay below 1.25× speedup. Passing the workload does not ensure a useful reduction in runtime.
Main results ↗A benchmark for application optimization
AccelerateBench.
Evaluating Agents for
Application-Level Program Optimization
Can coding agents make whole applications faster?
Real source trees. End-to-end workloads.
Correctness before speed.
Tencent Hy TeamHKUSTHKUST(GZ)THUPKU32CPU optimization
tasks
tasks
8Application
areas
areas
12Coding agents
evaluated
evaluated
8hInteraction budget
per run
per run
01 / Results
Download reported results ↓Measure the gain.
Keep the program correct.
One selected outcome per agent and task. Every score rewards faster execution only after the program passes its correctness checks.
Model performance
32 tasks · Reported in the paper
| # | Agent model | Effort |
|---|
Paper Table 1 · 12 agents × 32 tasks = 384 selected outcomes. Score@1 and Aτ are percentages; GM is in ×. Each agent pairs a model with a harness. An incorrect or unaccelerated outcome receives zero runtime-saving reward.
How to read these metrics
Score@1 averages correctness-gated runtime savings across 32 equally weighted tasks: C × max(0, 1 − candidate time / baseline time). A correct 2× speedup earns 50%; a correct program with no speedup earns zero.
Aτ is the fraction of tasks that are correct and reach at least τ× speedup. GM is the geometric mean speedup, with incorrect or unaccelerated outcomes assigned 1×.
For the eight configurations with repeated attempts, the median-reward run among the latest three policy-eligible scored attempts represents each agent–task pair. GPT-6 Astra, Gemini-3.8-Flash, Kimi K3 (max), and GLM-5.3 (max) use their sole attempt. All main-table metrics refer to the same selected record. The separate analysis cohort contains 480 runs.
02 / The benchmark
Optimize the application.
Preserve the computation.
Agents decide where and how to optimize. The task fixes what the application must compute, what can change, and where timing begins and ends.
01
Inspect & diagnose
Start from a working implementation, workload, and resource budget.
02
Edit & experiment
Coordinate source changes, compilation options, and runtime configuration.
03
Test & select
Use public correctness and performance feedback to retain a candidate.
04
Verify & score
An independent verifier rebuilds the final program and checks private inputs.
03 / Task collection
Scientific software, from simulation to I/OExplore the task collection.
Browse all 32 tasks. Open a task to see its resource limits and each agent's reported runtime savings.
32 tasks
Names and resource allocations follow the final paper. HPCG-derived, HPL-derived, and IO500-derived are adapted tasks, not official competition benchmark scores.
Scientific methods behind the workloads
04 / What the experiments reveal
Inside the optimization process.
Correctness, configuration, and interacting changes shape the final result. Three findings from the experiments.
The same source can run slow or fast.
In the controlled GROMACS replay, changing only build and runtime settings changes speedup across this range.
Controlled experiments ↗One change can depend on another.
A QiboTN code change brings almost no gain alone, but a substantial gain with another optimization in place.
Interacting code changes ↗480 runs / five agents / three runs per task
Look inside the optimization process.
Retained trajectories connect what agents inspect, build, test, repair, and submit with the final outcomes.
View the complete 480-run trajectory atlas
05 / Paper & authors
Read the paper Accelerate Bench
Evaluating Agents for Application-Level Program Optimization
‡ Project Lead † Corresponding author: chennuo26@gmail.com
Contact: Bowen Liu · bliu690@connect.hkust-gz.edu.cn
- Tencent Hy Team
- The Hong Kong University of Science and Technology
- The Hong Kong University of Science and Technology (Guangzhou)
- Tsinghua University
- Peking University
Cite this work
@misc{liu2026acceleratebench,
title = {Accelerate Bench: Evaluating Agents for
Application-Level Program Optimization},
author = {Liu, Bowen and Li, Yuhan and Chen, Nuo and
Luo, Weile and Peng, Miao and Zhang, Qintong and
Zhang, Qifan and Li, Jia and Gao, Ziqi and Huang, Lydell},
year = {2026},
url = {https://acceleratebench.com/}
}Resources
Build on Accelerate Bench.
The paper and reported results are available here. Versioned task packages, public tests, and verifiers are being prepared for release.