As AI agents take on more of model development, what should guide the improvements they pursue? RSI Arena studies a step toward recursive self-improvement with human feedback (RSIHF): agents carry out model development, while people help define what counts as improvement. Human judgments enter both the evaluation of a model and the next round of its development.

Ten frontier models each post-trained the same base model for six days on a shared cluster of 64 GPUs. People then compared the resulting assistants in a blind arena, and each agent received the comparison records and votes involving its own model for three more days of development. We evaluated both stages with human comparisons, capability benchmarks and safety tests.

Every agent produced a working assistant. The strongest, trained by GPT-6 Astra, received a human-preference rating close to NVIDIA’s own post-trained release of the same model after adjustment for answer length and formatting, with the NVIDIA reference evaluated with reasoning disabled. After three more days of development with access to people’s votes, eight of the ten models improved their style-adjusted rating relative to that reference. Benchmark gains were modest overall, and some agents encouraged longer answers.

People judged the models in a blind arena, and their votes informed Stage 2. The agents chose their own data, training code and checkpoints within a human-operated platform: people maintained the infrastructure, enforced budgets and could inspect recorded actions or terminate sessions. The last section discusses how human feedback can guide further model development.

A step toward RSIHF

Recursive self-improvement is the idea that an AI system could build a better AI system, which could in turn build a better one.1, 2 AI developers are working toward it: OpenAI aims for an automated AI researcher by 2028,3 and Anthropic reports that its models already lead about a quarter of its AI research and development.4 How close this is depends on whether AI agents can build models end to end, at the scale of real model development.

Post-training is a concrete step of this kind. It is the work that turns a pretrained base model into an assistant: a team chooses data, writes or adapts training code, runs supervised fine-tuning and preference optimization, and checks the results.13, 14 Most earlier tests of AI agents on machine learning work use tasks of a few hours on one or a few GPUs, scored automatically.6–10 The closest, PostTrainBench,11 asks agents to post-train small models for one named benchmark on one GPU for 10 hours. Real post-training takes many GPUs and many rounds of training and evaluation, and the models it produces are built for people. So in RSI Arena, people judge the models as well as benchmarks.

How RSI Arena works

Stage 1Train from the base model144 hours

1Base modelNemotron 3.5 Lightning 30B‑A3B
2Ten AI agents train iteach on its own, with 1,000 GPU‑hours and $300 of API credit
3Final modelsone checkpoint per agent
420 benchmarksHuman ArenaEvaluatepeople vote on blind pairs
each agent receives the votes on its own model

Stage 2Learn from human feedback72 hours

5Feedbackthe battles its own model played
6Agents keep trainingeach from its own Stage 1 model, with 500 GPU‑hours and $150
7New modelsone checkpoint per agent
820 benchmarksHuman ArenaEvaluate againNVIDIA Instruct as the common reference
The two stages of RSI Arena.

Stage 1: post-training

Ten frontier models took part as agents: GPT-6 Astra, MiMo-V2.6-Pro, Grok 4.7, DeepSeek V4.1 Flash, GLM-5.3, Kimi K3, Gemini 3.8 Flash, Muse Spark 1.3, MiniMax M3.1 Flash and Step 5 Preview. All ran in the same harness, built on the Codex agent loop,12 with the same instruction. Each started from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base,17 a mixture-of-experts model with about 30 billion parameters, and had 1,000 GPU-hours and $300 of API credit for its own model calls. Two agents joined on the third day, with the full budget.

The agents shared one cluster of 8 GPU nodes and 64 GPUs. Each could hold up to 16 GPUs at a time, so the agents waited in queues and lost preemptible jobs, as researchers do in a shared lab. The instruction asked each agent to train the strongest model it could and said that the final model would be evaluated on held-out data and by people. It did not prescribe data, methods or benchmarks, and it forbade training on test data. An agent submitted a model by nominating a checkpoint, a saved copy of the model, as often as it liked; its last accepted nomination became its final model. Stage 1 lasted six days, and a livestream showed every agent’s work as it happened.

Evaluation

We evaluated each final model on 20 benchmarks, 16 for capability and 4 for safety. We chose them before Stage 1 ended, and the agents never saw them. Our reference is NVIDIA Instruct, NVIDIA’s own post-trained release of the same base model, evaluated with reasoning disabled. To summarize the benchmarks, we measure how much of the gap between the base model and NVIDIA Instruct a model closes on each one, and take the median over the 12 capability benchmarks where Instruct leads the base by more than 10 points.

People judged the models in the Human Arena, on site at COLM 2026 in San Francisco and online. A visitor asks a question, two anonymous models answer side by side, and the visitor picks the better answer, a tie or “both are bad”. We fit ratings to the votes with the Bradley–Terry model,18 as Chatbot Arena does.19 People tend to prefer longer and more formatted answers,20, 21 so our main rating controls for answer length and formatting, as Arena.ai’s leaderboard does by default (style control).22

Stage 2: learning from people

After a release, model developers improve the next version from their users’ ratings and preferences. Stage 2 asks whether an agent can do the same. Each agent received the arena battles its own model had played, with the question, both answers and the vote, and had 72 hours, 500 GPU-hours and $150 of API credit to improve its model. The new models then went through the same 20 benchmarks and a second Human Arena, with NVIDIA Instruct again as the reference.

What happened in Stage 1

Every agent produced a working assistant. This was not a given: the base model had never learned to end its turn, so every agent first had to teach its model to stop. The best model, trained by GPT-6 Astra, closed a median of 76% of the base-to-Instruct gap across the 12 benchmarks. After adjustment for answer length and formatting, its estimated rating was close to NVIDIA Instruct’s, at 1,095 against 1,094. Without that control, people rated NVIDIA Instruct higher, at 1,182 against 1,161.

Human preference

Best agent model 1,095

NVIDIA Instruct 1,094; 795 accepted non-web votes

GPT-6 AstraAstra1,0951,161
NVIDIA InstructNVIDIA Instruct1,0941,182
Grok 4.7Grok1,0381,043
MiMo-V2.6-ProMiMo1,025992
GLM-5.3GLM1,0241,064
MiniMax M3.1 FlashMiniMax1,002983
Kimi K3Kimi982962
Gemini 3.8 FlashGemini981967
Muse Spark 1.3Muse Spark938913
Step 5 PreviewStep934936
DeepSeek V4.1 FlashDeepSeek887796

Rating with style control (Elo scale)

rating with style control95% bootstrap interval

Intervals use 1,000 battle bootstrap resamples. NVIDIA Instruct uses reasoning off; the right column gives ratings without style control.

Each model's battles

213–276 battles per model

How the votes went; each agent got these battles as feedback in Stage 2

GPT-6 AstraAstra5823213
Grok 4.7Grok4730267
MiMo-V2.6-ProMiMo3736263
GLM-5.3GLM4531236
MiniMax M3.1 FlashMiniMax3844257
Kimi K3Kimi3938264
Gemini 3.8 FlashGemini3437276
Muse Spark 1.3Muse Spark2752232
Step 5 PreviewStep2649245
DeepSeek V4.1 FlashDeepSeek2660230

wontieboth badlost

Includes web-search battles. Bar numbers are percentages of each model’s battles.

Human ratings in Stage 1: each model’s rating with style control (the tick) and its 95% interval, its rating without style control (right column), and how its battles ended. These battles became each agent’s feedback in Stage 2.

The agents used similar recipes. All ten fine-tuned on public instruction data, and six then ran preference optimization.16 Six trained LoRA adapters,15 small sets of added weights, and four trained all weights. Much of their time went to making training run at all. GLM-5.3 gave up on a standard trainer after about seven hours and wrote its own, about seven times faster. One of Kimi K3’s 8-GPU jobs waited 30 hours in the queue.

Spending more did not mean a better model. Gemini 3.8 Flash stopped itself after 12 hours and 19 GPU-hours. People rated its model above those of Muse Spark 1.3 and DeepSeek V4.1 Flash, which used about 1,000 GPU-hours each.

Progress toward NVIDIA Instruct

76% of the gap, at best

Median over 12 benchmarks of the share of the base-to-Instruct gap closed

GPT-6 AstraAstra76%
MiniMax M3.1 FlashMiniMax57%
GLM-5.3GLM52%
Grok 4.7Grok47%
Muse Spark 1.3Muse Spark45%
MiMo-V2.6-ProMiMo42%
Gemini 3.8 FlashGemini41%
DeepSeek V4.1 FlashDeepSeek41%
Kimi K3Kimi35%
Step 5 PreviewStep32%

median25th to 75th percentile

NVIDIA Instruct uses reasoning off. These 12 benchmarks have a base-to-Instruct gap of more than 10 points.

Each benchmark

10 of 120 scores above NVIDIA Instruct

0 = base model, 100 = NVIDIA Instruct; agents in the order above

AstraMiniMaxGLMGrokMuse SparkMiMoGeminiDeepSeekKimiStep
GPQA Diamond61666054−839441643−24
MMLU-Pro116831089112268103798885
SuperGPQA42605833234451453437
AIME 202691181843345−183936−18−18
HMMT Feb 20261313−7574−720720
LiveCodeBench v64228−1230563335−1926−7
SciCode113108121106096555810070
MRCR v28295−12791096699959362
IFBench69556251354536513660
Arena-Hard v293444738233737261226
MultiChallenge1610−221315149−234
WildBench v294818384587977606165

below the base modeltoward and past NVIDIA Instruct

Bold: above NVIDIA Instruct.

Benchmark progress in Stage 1 on the 12 capability benchmarks, where 0 is the base model and 100 is NVIDIA Instruct.

Benchmarks and people agreed only in part. Benchmark progress and the style-controlled human rating rank the ten models similarly, with a Spearman correlation of 0.72. Chat benchmarks scored by an LLM judge23 agree most closely with these style-controlled human ratings, with WildBench24 at 0.88, and competition mathematics does not. The model with the best AIME 2026 score25 placed eighth of ten in human evaluation. Length matters a great deal to voters. The longer answer won 72% of the votes that picked a winner, and controlling for style narrows the range of the agents’ ratings from 365 to 208 points.

Longer answers win

72% of decisive votes

Share won by the longer answer among 999 decisive votes between answers of different lengths

1–1.2×1.2–1.4×1.4–2×2–2.8×2.8–4×4–6.7×≥6.7×
1801492201388889135

Longer answer’s length over the shorter’s (×); votes in each group below

share won by the longer answer95% intervalcoin flip

Controlling for style

Agent rating range 365 → 208 points

How much style control moved each model’s rating

DeepSeek V4.1 FlashDeepSeek+91
MiMo-V2.6-ProMiMo+33
Muse Spark 1.3Muse Spark+24
Kimi K3Kimi+20
MiniMax M3.1 FlashMiniMax+18
Gemini 3.8 FlashGemini+14
Step 5 PreviewStep−2
Grok 4.7Grok−5
GLM-5.3GLM−40
GPT-6 AstraAstra−66
NVIDIA InstructNVIDIA Instruct−88

Rating with style control minus without

raised by style controllowered by style control

Answer length in the Stage 1 arena: the share of decisive votes won by the longer answer, by how much longer it was, and how much style control moved each model’s rating.

Safety was not part of the task, and some models got worse. The instruction never mentioned safety. On StrongREJECT,26 which measures how much useful help a model gives with harmful requests, three models gave more harmful help than the base model.

What happened in Stage 2

Eight models improved relative to the common reference. Eight of the ten models improved their style-controlled rating relative to NVIDIA Instruct, with a median gain of 28 points across all ten. We compare each model’s gap to NVIDIA Instruct within each arena, rather than directly comparing ratings from separately fitted rounds. The model trained by DeepSeek V4.1 Flash gained most, from 207 points below NVIDIA Instruct to 35 below, as its median answer grew from 61 to 508 tokens. With style control, the Astra-trained model and NVIDIA Instruct again have close ratings, at 1,061 and 1,054. Without it, NVIDIA Instruct is still first.

Stage 2 human preference

Best agent model 1,061

NVIDIA Instruct 1,054; 1,096 accepted non-web votes

GPT-6 AstraAstra1,0611,041
NVIDIA InstructNVIDIA Instruct1,0541,134
MiniMax M3.1 FlashMiniMax1,0301,032
DeepSeek V4.1 FlashDeepSeek1,0191,053
Grok 4.7Grok1,0081,002
GLM-5.3GLM9931,001
Step 5 PreviewStep992979
Gemini 3.8 FlashGemini986967
MiMo-V2.6-ProMiMo971943
Muse Spark 1.3Muse Spark958920
Kimi K3Kimi929929

Rating with style control (Elo scale)

rating with style control95% bootstrap interval

Intervals use 1,000 battle bootstrap resamples. NVIDIA Instruct uses reasoning off; the right column gives ratings without style control.

Gap to NVIDIA Instruct

8 of 10 narrowed the gap

Stage 1 to Stage 2 change relative to NVIDIA Instruct; style-controlled median +28 points

DeepSeek V4.1 FlashDeepSeek+172
Step 5 PreviewStep+98
MiniMax M3.1 FlashMiniMax+68
Muse Spark 1.3Muse Spark+60
Gemini 3.8 FlashGemini+45
Grok 4.7Grok+10
GLM-5.3GLM+9
GPT-6 AstraAstra+6
Kimi K3Kimi−13
MiMo-V2.6-ProMiMo−14

Change in the gap to NVIDIA Instruct (rating points)

narrowedwidened

Each arena is fitted separately. Each gap subtracts NVIDIA Instruct’s rating in that same arena.

Human ratings in Stage 2, and how much each agent model’s style-controlled gap to NVIDIA Instruct changed from Stage 1 to Stage 2.

The agents read the feedback closely. We read every agent’s Stage 2 record. Most learned what the votes say about substance, such as giving complete answers, refusing harmless requests less often and avoiding loops and cut-off answers. Some also acted on what the votes say about surface, as the discussion below describes. We found no evidence in the recorded Stage 2 actions of attempts to persuade voters, impersonate other models or manipulate the arena’s mechanics.

Benchmarks changed little. Six of the ten agents raised their median benchmark progress, and the median change was 2 points of the gap. Changes on benchmarks and changes in human ratings relative to NVIDIA Instruct showed little association across the ten models, with a correlation of −0.19. One model lost 18 points of benchmark progress, mostly because many of its answers no longer ended in the format the scorers read. That model also gives far more harmful help on StrongREJECT than before: 24.5, against 2.6.

The agents left much of their budget unused. They used 2,880 of their 5,051 GPU-hours and $541 of their $1,500 of API credit. Across both stages, how much an agent spent did not reliably predict how people rated its model. API cost and the Stage 2 rating have a correlation of 0.27, which with ten agents could be chance.

Human rating and API cost

Spearman ρ = 0.27

Stage 2 style-controlled rating against the agent’s API cost over both stages

Human rating, Stage 2

$10$20$50$100$200$5009409801,020NVIDIA Instruct 1,054AstraMiniMaxDeepSeekGrokGLMStepGeminiMiMoMuse SparkKimi

API cost over both stages (US dollars, log scale)

NVIDIA Instruct

API cost over both stages against the Stage 2 human rating.

Discussion

How far this is from recursive self-improvement

All ten agents independently selected data, implemented training and nominated checkpoints, producing working assistants within a human-operated research platform. The strongest model’s style-adjusted human rating was close to NVIDIA Instruct’s, with reasoning disabled for the reference. By the end of Stage 2, the best median benchmark progress reached 87% of the base-to-Instruct gap. This demonstrates that sustained, agent-directed post-training is feasible. It does not yet establish reliable recursive self-improvement: the experiment used one base model and one run per agent, and the agents trained a separate, smaller model while their own underlying models stayed unchanged. Repeated runs, different base models and experiments in which improved models conduct subsequent research are needed to test reliability and close the loop.

What agents learned from human feedback

The records show agents responding to incomplete answers, unnecessary refusals, repetition and premature termination, and using that evidence to revise their data, prompts and training. They also responded to presentation preferences. One agent concluded that length was “the only lever” on votes and adjusted its system prompt and training answers to encourage longer responses. Another selected prompts and checkpoints using a reward model that, by its own measurement, agreed with votes about as well as answer length alone. These observations show both the value of feedback and the difficulty of interpreting it: agents can identify problems that matter to users, but they can also pursue features associated with favorable ratings without establishing broader improvements. This is part of the wider challenge of optimizing against an imperfect measure.5

Human preference and benchmark capability

Benchmark progress and style-controlled human ratings were associated in Stage 1, with a correlation of 0.72, but the association was weaker in Stage 2, at 0.30. Changes in the two measures also showed little association across the ten models (−0.19). The DeepSeek-trained model gained 172 points in human rating relative to the common reference and 4 points of benchmark progress. Chat benchmarks scored by an LLM judge remained closer to human ratings, with WildBench at 0.79. Stage 2 also provided additional development time and resources and used a separate human arena, so these comparisons do not isolate the effect of human feedback. They show why both forms of evaluation are useful: benchmark gains and changes in user preference capture different aspects of model improvement.

Safety requires explicit attention

The task did not specify safety requirements, and outcomes moved in both directions. In Stage 1, three models had higher harmful-compliance scores on StrongREJECT than the base model. In Stage 2, one model’s score rose from 2.6 to 24.5, while other models improved on the same measure. These changes do not establish uniform improvements or regressions across all aspects of safety. Gains in human preference do not by themselves establish safer behavior, so safety needs explicit training objectives and evaluation independent of preference and capability.

Using compute effectively

Compute use showed little association with human ratings across the ten Stage 1 runs (−0.21). Gemini 3.8 Flash used 19 GPU-hours and was rated above two models whose agents used about 1,000 GPU-hours each. In Stage 2, of the two agents that exhausted their GPU budgets, one improved on benchmarks and the other lost the most benchmark progress. These observations do not establish that compute is unimportant; a larger allowance alone did not ensure better outcomes here. Engineering failures and narrow internal tests point to training reliability and checkpoint evaluation as practical ways to make better use of resources.

Records that people can check

Recorded trajectories reveal decisions that final scores alone would miss. Two agents found benchmark items in their training data and removed them without being asked. Independent checks also found discrepancies in some agent-written model cards, including training counts and reported hardware. These findings support checking agents’ accounts against execution records and submitted artifacts. The released checkpoints and documented trajectories provide a basis for examining how development decisions affect model behavior.

Toward recursive self-improvement with human feedback

We view RSI Arena as a step toward recursive self-improvement with human feedback (RSIHF): agents carry out model development while users’ judgments help guide what to improve. Feedback can identify problems in use, guide priorities for further training and help assess whether users prefer the resulting changes.

Votes remain partial measurements of what people value. Human ratings, benchmark performance and safety did not consistently improve together, and some preference gains accompanied safety regressions. Research on learning from human feedback also shows that favorable judgments need not imply better answers.27 Independent capability and safety evaluations remain necessary alongside user feedback.

The study establishes feasibility more clearly than reliability. Future work should test consistent improvements across repeated runs, different base models and successive rounds of human feedback.

How to obtain sufficiently informative feedback as autonomous development grows remains an open question. Each stage gathered roughly 1,300 accepted votes, but this experiment does not establish how much feedback reliable improvement requires. Further work should examine when agents should seek human input, how they interpret it and how automated evaluators can remain calibrated to users over successive rounds of development.

Explore RSI Arena

This edition of RSI Arena has ended, and so has its livestream. The results and daily reports are at rsiarena.org, and the submitted model checkpoints are available on Hugging Face for further study. We look forward to seeing you at the next one.

We thank everyone who voted in the Human Arena, at COLM 2026 and online, and our collaborators at Hugging Face, Scale AI, the University of Notre Dame, the University of Washington and Stanford University.

References

Recursive self-improvement and AI research

  1. Irving John Good. Speculations Concerning the First Ultraintelligent Machine. Advances in Computers, 1966.
  2. Jürgen Schmidhuber. Gödel Machines: Fully Self-Referential Optimal Universal Self-Improvers. In Artificial General Intelligence, Springer, 2007.
  3. Rebecca Bellan. Sam Altman says OpenAI will have a ‘legitimate AI researcher’ by 2028. TechCrunch, 2025.
  4. Marina Favaro and Phillie Wright. Measurements for understanding the pace of AI development inside frontier labs. Anthropic, 2026.
  5. Yue Huang et al. Reward Hacking Challenges Oversight of Autonomous Research Agents. 2026.

AI agents for machine learning work

  1. Jun Shern Chan et al. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. 2024.
  2. Hjalmar Wijk et al. RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts. 2024.
  3. Qian Huang, Jian Vora, Percy Liang and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. ICML, 2024.
  4. Giulio Starace et al. PaperBench: Evaluating AI’s Ability to Replicate AI Research. 2025.
  5. Chris Lu et al. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. 2024.
  6. Ben Rank et al. PostTrainBench: Can LLM Agents Automate LLM Post-Training? 2026.
  7. OpenAI. Codex CLI. 2025.

Post-training

  1. Long Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022.
  2. Nathan Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024.
  3. Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
  4. Rafael Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS, 2023.
  5. NVIDIA. NVIDIA Nemotron 3.5 Lightning 30B-A3B. Model card, 2026.

Evaluation by people and by benchmarks

  1. Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 1952.
  2. Wei-Lin Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ICML, 2024.
  3. Prasann Singhal, Tanya Goyal, Jiacheng Xu and Greg Durrett. A Long Way to Go: Investigating Length Correlations in RLHF. 2023.
  4. Yann Dubois, Balázs Galambosi, Percy Liang and Tatsunori B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. 2024.
  5. Tianle Li, Anastasios Angelopoulos and Wei-Lin Chiang. Does Style Matter? Disentangling Style and Substance in Chatbot Arena. LMSYS Org blog, 2024.
  6. Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks, 2023.
  7. Bill Yuchen Lin et al. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild. ICLR, 2025.
  8. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović and Martin Vechev. MathArena: Evaluating LLMs on Uncontaminated Math Competitions. 2025.
  9. Alexandra Souly et al. A StrongREJECT for Empty Jailbreaks. NeurIPS Datasets and Benchmarks, 2024.
  10. Jiaxin Wen et al. Language Models Learn to Mislead Humans via RLHF. 2024.

Cite this work

Please cite the RSI Arena paper on arXiv when referring to these experiments or results.

BibTeX
Download .bib
@misc{huang2026rsiarenarecursiveselfimprovement,
      title={RSI Arena: Towards Recursive Self-Improvement with Human Feedback},
      author={Yue Huang and Yanbo Wang and Lichi Li and Yuanyuan Chen and Andreas Haupt and Xiaonan Luo and Zheyuan Liu and Jihye Choi and Felix Juefei-Xu and Xiangliang Zhang and Radha Poovendran and Misha Sra and Ying Liu and Ben Burtenshaw and Thomas Wolf and Alex Pentland and Zichen Chen and Zhangchen Xu},
      year={2026},
      eprint={2610.33624},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.33624},
}