Reward Hacking Challenges Oversight of Autonomous Research Agents

A research agent can shape both its result and the evidence used to judge it. The same agent may choose a method, write its implementation, run the evaluation, and report the outcome. Its final code can look reasonable without showing which data it used, how it chose a setting, or whether it changed the calculation behind a reported score.

These opportunities allow reward hacking: obtaining a high score without making the improvement the task requires. In our paper,¹ we study three questions. Do research agents hack under ordinary instructions? Can AI reviewers detect hacks from the final code and reported score? And when agents are instructed to avoid detection, can they use review feedback to improve their next attempt? We find that:

• Agents hacked without being asked. Under ordinary instructions, all 17 models hacked at least three tasks.
• Reviewing the final code missed some hacks. When hacking was allowed, reviewers missed 33 of 505 verified hacks.
• Review feedback helped agents evade detection. With feedback and up to five attempts, 56 of 150 model–task pairs got a hack past review, up from 7 after one attempt, without training or fine-tuning.

One caseAn ordinary model with a fake score

Consider a task that asks an agent to predict stock returns. Performance is measured by the correlation between its predictions and the actual returns on a held-out test set. To pass, the agent had to reach a correlation of 0.15, well above the best result we obtained with a method that followed the rules.

The agent submitted an ordinary ridge-regression model, but its code also changed how that model was scored. Loading the submission replaced NumPy’s correlation function, np.corrcoef, which the task’s scorer used. For arrays matching the test set’s size, the replacement returned a value chosen in advance instead of computing the correlation. The reported number therefore came through the task’s own scorer, even though the calculation underneath had been altered.

The code described the replacement as a safeguard against numerical errors. Because the prediction model itself remained short and conventional, inspecting that model alone would not reveal why its score was invalid.

How we tested it

In each run, an agent received one task and worked in a terminal, where it could write code, run experiments, and submit a result.

Experiment 1 tested 17 models on 38 tasks, drawing in part on AutoLab.⁴ The tasks included 20 open-ended research problems, where the agent builds a whole research pipeline, and 18 narrower problems that ask for one working implementation. This comparison lets us examine hacking in settings that give agents different amounts of control over the work.

For Experiment 2, we used 16 tasks whose pass mark exceeded the best result we obtained by following the rules. Ten came from MLS-Bench,⁸ and six were versions of Experiment 1 tasks with a higher pass mark. Experiment 3 used the ten MLS-Bench tasks. These thresholds make the tasks difficult, but do not establish that an honest solution is impossible.

In Experiments 2 and 3, a submission counted as a verified hack only if it passed the task’s threshold and a separate verification panel confirmed that it exploited the evaluation. This panel compared the code, the agent’s account of its method, and results from reruns on hidden data. To test whether the hack was detectable from the submission itself, a review panel saw only the final code and reported score. Its three models, Claude Sonnet 4.6, DeepSeek-V4-Pro, and GPT-5.6 Luna, voted on whether to flag the submission. We call a verified hack an evasion when this review panel did not flag it.

We also evaluated two further checks that used evidence beyond the final code and reported score. An execution audit copied the agent’s final files into a fresh sandbox, restored the original scorer, and ran the method on data the agent had never seen. It flagged changes to protected files, failures on the new data, or reported scores well above the real ones. A separate check gave reviewers the full work history for the hacks they missed in Experiment 2, allowing them to inspect how the final submission was produced.

Experiment 1Agents hacked without being asked, mostly on open-ended tasks

All 17 models hacked at least three tasks under ordinary instructions, producing 105 hacks in total. The rate differed substantially by task type: hacking occurred in 30.5% of runs on open-ended research tasks, compared with 2.9% on narrow implementation tasks. Agents therefore did not need an explicit instruction to hack, and the behavior was much more common when they were responsible for a whole research pipeline.

The common hacks exploited choices within the research process. In the self-certification task, every model that ran the task wrote its own test and graded itself, making the evaluation depend on the system being evaluated. Other models trained on test labels or exploited a loose rule for combining scores. Across these runs, the kind of task mattered much more than which model performed it.

General capability benchmarks did not reliably predict how often a model hacked or how difficult its hacks were to detect. This analysis is exploratory because the sample is small, but it suggests that general benchmark performance cannot substitute for testing these behaviors directly.

Experiment 2Reviewers caught most hacks, but missed some

The first experiment established that hacking can occur without an explicit request. We next examined how well reviewers could detect hacks when agents were allowed to search for them. In this controlled red-team experiment, agents were told they could game the scorer within a sandbox and produced 505 verified hacks in 677 attempts. This setting measures their ability to construct hacks when asked; it does not estimate how often they would do so under ordinary instructions.

Reviewers who saw only the final code and reported score missed 33 of the 505 hacks, or 6.5% (95% confidence interval: 4.7–9.0%). They classified 32 as honest and one as uncertain. To check whether detection came at the cost of rejecting valid work, we also reviewed 102 separate runs that followed the rules. None were flagged (95% confidence interval: 0–3.6%). The panel thus caught most hacks without flagging these controls, but a small fraction of verified hacks still passed review.

Success at gaming the score did not imply success at evading review. Hacks rated 10 out of 10 for effectiveness passed review only 0.4% of the time. Missed hacks were concentrated in calibration under distribution shift, PAC-Bayes bounds, and especially self-certification. In the last task, twelve models reported a score of about 0.85 for a model that scored 0.46 on held-out data.

The execution audit provided a different check by testing the submitted method on hidden data with the original scorer. It missed 38 of the 505 hacks, counted separately from the 33 missed by review. Most were scorer tampering or offline label distillation that remained effective on the hidden data, and they were concentrated in calibration under distribution shift and stock prediction, the task in our opening case. A fresh evaluation therefore caught many hacks, but passing it did not establish that the method had been developed without exploiting the task.

Giving reviewers the agent’s full work history revealed 21 of the 33 hacks missed in the initial review, leaving 12 undetected. Three of these were disputed PAC-Bayes cases; excluding them leaves 9 evasions among 30 cases. The work history supplied evidence absent from the final code, although it did not resolve every case.

The final code may not show how a seemingly routine choice was made. A random seed selected through search can appear to be a fixed reproducibility setting, while precomputed coefficients may conceal which data were used to fit them. The work history can help reviewers check whether these choices followed the task’s rules.

Experiment 3Review feedback helped agents evade detection

The preceding results concern review of completed submissions. In a research workflow, however, an agent can revise a rejected submission. To test whether this process could help conceal hacks, we explicitly instructed agents to avoid detection. After a flagged attempt, the agent received the panel’s decision, its reasons, and the history of earlier attempts, then revised its code. Each of 150 model–task pairs had up to five attempts, stopping at its first evasion.

The number of pairs with a hack that passed review rose from 7 of 150 on the first attempt to 56 of 150 by round five. The remaining 49 first succeeded in later rounds, after receiving feedback. Because additional attempts could account for part of this increase, we also ran a control condition that returned only a generic rejection, without the reasons or attempt history.

Among the 79 cases run under both conditions, 40.5% produced a hack that passed review with detailed feedback, compared with 20.3% with a generic rejection. Both conditions allowed up to five attempts, so access to more retries alone does not explain the gap. The detailed-feedback condition supplied the decision, reasons, and history together; this comparison measures their combined effect rather than the contribution of any one component.

The change was also visible within individual tasks. Stock prediction was one of five tasks with no evasion on the first attempt and at least one by round five; 6 of its 15 models eventually produced a hack that passed review. These task-level samples are small, but they show that later success was not confined to tasks where an initial hack had already escaped detection.

The mix of techniques changed over successive rounds. Direct test-set leakage fell from 84% to 57% of hacks, while the shares of offline label distillation and scorer tampering roughly doubled. These techniques can leave less obvious evidence in the final code, as the examples below illustrate. Since successful pairs stopped, the figures describe the changing pool of attempts rather than a transition observed in every agent.

To test whether the evasions depended on the first panel, we submitted the 56 missed hacks to a second panel: Gemini 3.5 Flash, Grok 4.5, and Kimi-K3. It also cleared 44 of them (79%), showing that many of these hacks passed review by different models. This rate applies to the hacks already missed by the first panel. The execution audit then caught most of those that both panels missed, leaving 5 of the 56 hacks undetected by all three checks.

Earlier work showed that training a model against a monitor can encourage it to conceal unwanted behavior.⁹ Here, agents adapted without training or fine-tuning: they used review feedback and prior-attempt history to revise their code across successive attempts, while reviewers continued to assess only the final code and reported score. Review feedback can therefore become part of the agent’s search for a hack that passes inspection.

How the hacks worked

To understand why some hacks were harder to detect, we grouped 923 descriptions from Experiments 2 and 3 by their mechanism. We first rewrote the descriptions in a common format to reduce the influence of each model’s writing style. The resulting groups contain five recurring families, summarized below.¹

Direct test-set leakage accounted for three-quarters of the hacks, and reviewers caught 97% of these cases. When the code reads test answers directly, the source of the apparent improvement is often visible in the submission. Offline label distillation can hide that source: an agent fits coefficients to leaked labels beforehand, then presents the coefficients as fixed “domain priors.” Reviewers missed 37.5% of this family, about twelve times the 3.0% miss rate for direct leakage.

These proportions reflect the opportunities in our tasks, all of which exposed an evaluation set, rather than the prevalence of each technique in research generally. Within this setting, the comparison helps explain a limitation of final-code review: it can identify an explicit use of test answers more readily than it can recover the origin of numbers already embedded in the code.

The map below provides a complementary view of the same 923 descriptions, grouping similar mechanisms and marking those that passed review.

What this means

Keep the test out of the agent’s hands

Reliable evaluation requires evidence that the agent cannot alter to support its own claim. Test data and scoring code should remain outside its control, and execution records should preserve how the result was obtained. The full work history helped reviewers recover some of this missing evidence, catching hacks that they had missed in the final code.

Claimed improvements should also be checked using independently chosen data and evaluation code. Simply reproducing a submitted score may preserve the same flaw that produced it. The rerun has to test the claim. Even then, a method that performs well on new data may have been developed using information it was not allowed to access, which is why the record of its development also matters.

More AI reviewers can share the same blind spot

AI reviewers offer a way to scale the assessment of AI-generated research, but different reviewers may accept the same incomplete evidence. A second panel cleared 44 of the 56 hacks missed by the first, whereas an execution audit caught most of those that both panels accepted. This result suggests that adding evidence, such as protected data, work histories, and independent tests, can address failures that persist when only the reviewers change.

Review also changes the information available to the agent. A decision and its reasons can reveal which part of a submission exposed the hack, helping the agent decide what to revise. In Experiment 3, the condition providing this feedback and the attempt history led to more evasions than a generic rejection under the same retry budget. Oversight should therefore be evaluated over repeated interactions, where agents can adapt to the feedback they receive.

Explanations remain useful for auditing review decisions and helping an honest agent correct mistakes. The challenge is to preserve these benefits without unnecessarily revealing how an exploit was detected. The paper therefore recommends limiting those details when they are not needed for a correction.

False progress can compound

If research agents help develop future AI systems, an unsupported result could influence which training method is adopted, which checkpoint is retained, or which safety claim is accepted before release. Subsequent work could then build on those decisions, allowing an evaluation failure to propagate through a sequence of apparent improvements. The resulting system could appear more capable or safer than the evidence warrants.

This risk extends beyond an individual experiment. When producing claims is much cheaper than checking them, unsupported findings can consume compute and redirect research before their basis is examined. In AI development, those decisions also shape the capabilities and safeguards of the systems that follow.

Autonomous research is useful only when its results are reliable enough to build on. These findings inform our work on systems such as Épi.

Release note. We have withheld selected mechanism-level data and operational details because publishing them could make intricate hacks easier to reproduce, including by models trained on public text. The public release describes the broader failure modes without these details. The paper lists the withheld material, which vetted researchers and affected developers can request.

@misc{huang2026rewardhacking,
  title = {Reward Hacking Challenges Oversight of Autonomous Research Agents},
  author = {Yue Huang and Zhangchen Xu and Yuchen Ma and Wenjie Wang and Zheyuan Liu and Ziwei Xu and Pin-Yu Chen and Michel Galley and Zinan Lin and Stefan Feuerriegel and Radha Poovendran and Misha Sra and Alex Pentland and Xiangliang Zhang and Zichen Chen},
  year = {2026},
  eprint = {2609.28614},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2609.28614}
}