# « the era of (auto) research » — Elie Bakouch — session 2026-07-01T19:05:00.000Z → 2026-07-01T19:25:00.000Z _63 transcript lines · 37 slides · source: full recording_ ## Transcript So what building the wrong thing looks like if you've got velocity. Hey. Hi, everyone. Thanks for being here. Yeah. I'm super happy today to talk about, automated AI research and, especially, all those, like, frontier model. At, automated AI research task. So I'm Eli. I worked at Paramount Tech as a research engineer. And, yeah, I will go through our work on this subject. So first, I want to basically explain a bit why we are doing that and why we think it's super important to do that in the open. So first, I think we we all agree that, we've heard about, like, BigLab saying that this bat Thing called recursive self improvement is coming very soon. So recursive self improvement is like model training models, without, human intervention, basically. But we don't have any benchmark to basically quantify if this is true or not. Right? And even less, we don't have, like, a third party benchmark by non BigLabs to to to see if it's something coming soon or not. And the other parts Is that we think that, it's super important to understand all those model, do research because we think that a lot of the scientific research that will come into the coming years, will, be based also on AI tools. So it's super important to understand how those models do research, not just only AI research. So we try to build kind of this environment to test the capabilities of the model to do so. So it all started with Andre Carpathi, that's basically advanced by doing this video where he trained, not a GPT two from scratch in, like, ninety minutes. Like, GPT two training takes, like, weeks, and, no. In, two years ago, I think, it only took, like, ninety minutes. So what does it mean to reproduce GPT two in ninety minutes? It means that in ninety minutes, you achieve this target loss. And, yeah, and that's at this point You have the same loss than, GPT two. You consider that your model is somewhat of equal performance. Then what happened is that the community took this repo, this GitHub repo, and created another one called modded nano gbt. And this effort was led by someone called Keller Jordan. And what happened is that they basically took this ninety minutes, then forty five minutes, and then now we can train like GPT two validation loss model in less than two minutes, which is honestly crazy, and it took, like, two years to to achieve this. So it's a very strong benchmark where, a lot of very talented researchers iterated on. Yeah. So we decided to take this environment off speedrun. So where this is, it's kind of a game. So the goal of the game is to achieve this, loss in the few ways in the shortest amount of time. So this is the nano GPT-one. And you can, you don't have almost any constraints. The only constraint that you got is that you need to use the same validation and training data. Right? There is a new speedrun called the optimizer speedrun that was released, a few months ago, and here it's slightly different because, you can only change the optimizer, related parameters. So for instance, NanoGPT, you can change the architecture. Do MOE, do, attention, whatever, optimizer speedrun. You can only change, like, Adam to shampoo or whatever optimizer is your favorite. Yeah. And so this is a bit more research y because, it's less about optimizing the program to be as fast as possible, but more like finding the best method possible no matter the the the time you put into the computer. Right? So, yeah. Why it takes Speedvan as an environment for automated AI research? First, we think that it's a good evaluation. We'll see later why. And this is kind of the main focus of this talk. But we also think it's probably a good training environment because, it's a way to give the model a reward. So the reward is positive if the model beat the speedrun and, beat the last record, sorry. And the reward is Zero or negative if you didn't manage to do it. So it's a good environment to train model. It's also quite fast. Like, as you see, previously, we call around two minutes for the optimizer one. Each run take about, like, fifteen to twenty minutes. And, yeah. And there is, like, clear rules, basically. And we also think it's, like, a good environment to make discovery, so, like, kind of breakthrough into our research because, there is those clear A rule that you can verify or not. Yeah. So yeah. So what we did, so the release was, like, about two months ago, and, there was this optimizer speed run. And we decided to basically compete with the community by launching two AI agents, so codecs and Cloud Code. Codecs was like GPT 5.5 with Xi, and, Cloud Code was Opus 4.8. With x high. And, yeah, we decided to basically let the agent free on our cluster, and, and just iterate on it. So we have, like, v one, v two, v three is just basically us stopping the agent and then restarting v three. It was, like, one or two days before the release because we saw that our agents no longer have the best record. So we were like, okay, Take all the the human, record in the last few weeks and just try to To improve upon it, and and and it works. Yeah. And we also have this novelty track where the goal is to, beat the record with only novel IDs. And we'd see that this this was more complex for the the models. So RMS is very simple. Honestly, we could have just replaced it with slash goal, but the the there was no slash goal at the time, so we made our own goal. Md. It's actually quite fun that we choose the Name. And we had the go. Md and kind of agents. Md that defined the rules. And we let the agent propose IDs, and then it can submit a job, with sbatch on our Slurm cluster. And, basically the way it works is that it can submit on nodes that are available but only under a certain permission, which means that if someone want to use this node, the model just, like, cancel the job. It's called preemptible. Permission. So, yeah, then it measure the it read basically the training logs, then decide if it's a record or not. To validate a record, you need to basically pass a statistical threshold to make sure that it's just not seed optimization and it's just not on them. Right? So, yeah, a few results from this experiment. The first one that was honestly very painful to work with is that code, Claude Claude keeps stopping every nine or ten hours and basically say Yeah. I cannot improve the record. It's too hard for me. There is no way to to go beyond it. And then I was just like, okay. Continue, explore new direction, and just go again for ten hours and then say, yeah. I cannot beat the recording. So, basically, one third of the time, the Cloud Code agent was idle because I had no way to basically monitor it. And Codecs, totally the opposite, just worked for all the all the time. And, yeah, almost never idle, never asked for question, and and and very impressive in that way. We also give the option for the model to basically write a bunch of stuff into what we call a scratch pad, which is basically the active memory of the model. We observe that, basically, codecs write a lot on the scratch patch. So each plot that I will show are kind of normalized by the number of active So this is not only about codecs working more. It's really different behavior. So, yeah, you see that white's a lot more to this scratchpad, to this memory. And, the shape of the like, the the I don't know, the tone of each file was also super different. Like, Claude was super excited about getting new record with a bunch of emoji and so on. And Claudex was just like, here is what I do. Here's the decision I Like, what I would do next, like, super robotic, kinda. Yeah. We also have this plot where, basically, we saw that codecs was, spawning much more sub agents than clouds. We saw that codecs burned much more token than cloud. So I think in total, it was, like, kind of billions of token, but it's like there is obviously this input to input caching that make it it's not like 1,000,000,000 output token. So, yeah, we also see that codecs did a lot of compaction because it only had, like, two fifty ks, context window. And clouds only do it, like, one per hour, and codecs is more like, no, it's even less than one power for I mean, one for the full run for Cloud. And codecs was, like, one, was 20 every one hour. So, yeah. Yeah. Here is the main results. So what this plot shows Is that basically we so in in the white, you see that the the human recall progression. Right? And in red, you see cloud. I mean, it's supposed to be orange, but whatever. And in blue, you see codecs. Right? And you see that at almost every time, cloud and codecs are better than the human record, and cloud is super good at the beginning, very, very fast to achieve very good score. Yeah. And one thing That is super important is that the model have the ability to basically fetch the human records at any time, and that's what codecs did that's what Cloud did, sorry. Because when I restarted it, it basically fetched the new record from human and improved upon it. Yeah. So the result is that, I think at the time, the best record was, like, 2,990 steps, and we beat it by, like, 50 or 60 steps for Cloud, and the codecs was, like, 20 steps above. So I think it's both impressive. And yeah. So this is not released yet. This is something that we are working on currently. And basically, the idea is that this is a cool experiment to do, but it lack of structure. Right? If you want to do a real benchmark, you want to do multiple seed, you want to do, yeah, proper Thing where you you you basically put all the model and LNS in the same condition. Right? So this is what we are working on right now. And basically, the idea is to do three different track, one without any access to really like, measure the capability of the models to do AI research based on only the model weight knowledge, one with only archive paper, and one with, like, full access. So it also have access to the the, like The latest record by human. And for this, we plan to do both the NanoGPT track one, which is the original one, and the optimizer speedrun, where we only launch we only constrained the optimizer to be to be novel, basically. Yeah. So I will present some results on the optimizer speedrun. This is basically what we got. So we let the agent iterate for six day, almost, five days, let's say. And we see that, codecs, kmee, and clouds, are super effective. So for GLM, this is not finished run. Right? So the model is actually still iterating on the cluster right now. But we see that Claude is once again very good at it. And we see that, surprisingly, Kimi is also very competitive and kind of have this breakthrough on day four where he kind of beat codecs with a new record. Right? It's also interesting To see that Claude is much more progressive in the way it improves the recall, and Kimmy has really this step function where I kind of do a breakthrough and so on. So this is an interesting plot because, I mean, since there is quite a lot for an eval, but you can change this axis by also the number of output tokens. And then kind of tell a different story because cloud in max mode consumes so much more token than Codecs and, kimi. And you also see that kimi is actually super efficient, for the number of token that, each use. So it's kimi k 2.7 code. So yeah. We also see that they have a different approach to, using the literature and papers. So for instance, like, Claude is doing a lot of search on papers. And actually, Claude found a paper that no other model found. And Actually, lead to the best record, so it's kind of funny. And, yeah. One of the main issue of all of this is that, when I when I launched these these agents and I think that's something important that I want you to kind of, remember for this talk is that when I launched these these different agents, I was expecting them to come up with some crazy ideas on the optimizer that's, like, no One of Discover. But honestly, it wasn't the case. They did some clever trick where basically they combine different papers. They kind of do plus one improvement over a bunch of method, But there was really no novel optimizer or mechanism that was coming from those models. And I think that's kind of telling that even something that is not simple, but I'd say that it's kind of accessible for people. Right? For, like, human Researcher, spending, like, days and weeks for the the model, like, cannot, like, find new, optimizer and mechanism. So we believe that there is a way to basically make it more, make it better for discovery instead of evaluation. And this is coming from, this is very inspired from, Alpha Evolved by Google and also a bunch of paper that have been released since then. It's kind of this multi System that interact together, bunch of generator. You have a closed model, but you also have open source model here that are super effective for the cost. Right? They can suggest ideas, then you run the speedrun, so you get the reward. Then you have a judge that basically give a quality feedback. Can also be, like, the judge also have this taste. You can kind of have, like, the judge have a taste about the the method, if it's good or not. It's outside the loop. And then you can, basically decide which method you want to scale to a larger number of parameters and, number of token. So this is kind of the scale part of the speedrun because some a lot of methods in the speedrun community, people are often saying that they don't work at large scale. So I think it's very important to also put scale elements in this loop. And I think Also, that's, human are super useful here to basically judge the idea of agents, kind of steer them in the right direction, and so on. Yeah. So we didn't try it yet. I mean, we are kind of trying it right now. And, we hope that this will lead to to to new discovery in AI research, at least. And also a way is that you can define multiple speedrun. So this is the next slide. If you like, it's from Have blank slides, but if you if you don't have the reference, good for you. It means that that you're not too online. But the idea is that, by changing the objective and the constraints of the speedrun, you can basically create a lot of diversity and constrain the model to go into a certain direction and, yeah, and make those discovery. So, at PrimeIntellect, we are doing a bunch of stuff in this direction. There is a bunch of stuff here that we I mean, most of it we They don't release yet, but we are working on, GPU sandboxing to allow model to iterate into sandbox because you need GPU sandbox for this kind of stuff. We are working on our own agents that are very efficient for, like, LLM frameworks. So it means, like, you have the file system, and you can write information, read from it. And you also do, like, this programmatic tool coding thing. We are also training the model to be good at it on top of, like, open source model. And, the thing that we already released is that we have those set of library and product called verifier, primary, and hostel training where you can basically train, evaluate any environments on any RMS, and the model that you can train can be like JNAM 5.2, which is very big. And and, yeah, we have, like we work a lot on making those library very efficient to to ship the best quality for for our clients. Yeah. I mean, yeah, super About this domain. Once again, I think it's super important to have, a part of, like, this recursive self improvement to up to happen in the open because there is actually a lot of people working that are not on big labs. So you need to basically, yeah, make it easy for people to understand all those model work to do research and so on. So that's kind of our goal. And, yeah. Thanks a lot. ## Slides ### 00:00:56 # AI Engineer World's Fair - MERGE - twilio - Snor - INNGEST - Z.AI - BAND - neo4j - Google ### 00:01:19 # Why measure AI research capabilities THE ERA OF (AUTO) RESEARCH ### NO BENCHMARK EXISTS - No third-party benchmark for RSI. - Labs say it's coming, nobody is measuring it publicly. ### RESEARCHERS NEED ACCESS > "Scientific research is only possible when scientists have access to the best tools of the time - including AI-based tools." > FEI-FEI LI @DRFEIFEI ### WHY THIS MATTERS ## Engineering the future of AI ### 00:01:50 # Why measure AI research capabilities ## NO BENCHMARK EXISTS No third-party benchmark for RSI. Labs say it's coming, nobody is measuring it publicly. ## RESEARCHERS NEED ACCESS > "Scientific research is only possible when scientists have access to the best tools of the time – including AI-based tools." > FEI-FEI LI @DRFEIFEI ## Engineering the future of AI ### 00:02:16 # Why measure AI research capabilities ## No Benchmark Exists - No third-party benchmark for RSI. Labs say it's coming, nobody is measuring it publicly. ## Researchers Need Access > "Scientific research is only possible when scientists have access to the best tools of the time - including AI-based tools." > FEI-FEI LI @DRFEIFEI ## Engineering the future of AI ### 00:02:48 # Karpathy reproduced GPT-2 in 90 minutes GPT-2 took weeks in 2019. llm.c did it in 90 minutes for $20 ### 00:03:22 # 45 min -> 1.32 min in two years 84 open records public leaderboard anyone can beat. ### Key Milestones in GPT-2 Loss Reduction - **4 ### 00:03:47 # Two speedruns, two kinds of freedom ## THE FULL SPEEDRUN ### NANOGPT Change everything but the data. ## THE SUBTRACK ### OPTIMIZER Change only the optimizer. [Two line graphs comparing Method A and Method B for decreasing loss. The left graph shows loss over time, with Method B reaching GPT-2 LOSS faster. The right graph shows loss over steps, with Method B reaching TARGET 3.28 in fewer steps.] ### 00:04:17 # Two speedruns, two kinds of freedom ### THE FULL SPEEDRUN ## NANOGPT Change everything but the data. B is better: reaches the bar in less time ### THE SUBTRACK ## OPTIMIZER Change only the optimizer. B is better: reaches the bar in fewer steps ## Engineering the future of AI [Two line graphs comparing "METHOD A" and "METHOD B" for loss reduction over time/steps. Both show Method B reaching a target loss (GPT-2 LOSS or TARGET 3.28) faster.] ### 00:04:47 # Two speedruns, two kinds of freedom ## THE FULL SPEEDRUN ### NANOGPT Change everything but the data. B is better: reaches the bar in less time ## THE SUBTRACK ### OPTIMIZER Change only the optimizer. B is better: reaches the bar in fewer steps # Engineering the future of AI [Two line graphs comparing Method A and Method B for loss reduction over time/steps. The left graph shows a GPT-2 LOSS target, and the right graph shows a TARGET 3.28.] ### 00:05:17 # Why speedruns 1. Evaluation 2. Training environment 3. Discovery ## Engineering the future of AI ### THE ERA OF (AUTO) RESEARCH ### 00:05:49 # Why speedruns 1. Evaluation 2. Training environment 3. Discovery ## Engineering the future of AI ### 00:06:18 # Two agents, four waves - **01 V1** (BEAT 3500) Beat the Muon 3500-step baseline. - **02 Novelty** (NOVEL ONLY) Every idea must pass a novelty check. - **03 V2** (+ 3000) Seeded from best of v1. Push toward 3000. - **04 V3** (+ SUB-2900) Seeded from v2 + public PRs. Chase sub-2900. ### 00:06:51 # Engineering the future of AI ## Two agents, four waves ### 01 V1 (BEAT 3500) - Beat the Muon 3500-step baseline. ### 02 Novelty (NOVEL ONLY) - Every idea must pass a novelty check. ### 03 V2 (+ 3000) - Seeded from best of v1. Push toward 3000. ### 04 V3 (+ SUB-2900) - Seeded from v2 + public PRs. Chase sub-2900. ### 00:07:12 # Two agents, four waves ## 01 V1 Beat the Muon 3500-step baseline. ## 02 Novelty Every idea must pass a novelty check. ## 03 V2 Seeded from best of v1. Push toward 3000. ## 04 V3 Seeded from v2 + public PRs. Chase sub-2900. ### 00:07:45 # The harness: an autonomous research loop `goal.md / AGENTS.md / plan.md` define each wave. The agent loops on a shared GPU cluster: `~10k` runs, subagents, sweeps, novelty checks. ### DEFINED PER WAVE - **goal.md** - reach 3.28 in the fewest steps - **AGENTS.md** - the rules and the freedoms - **plan.md** - the agent's running plan ### THE LOOP - THE AGENT ITERATES ON A GPU CLUSTER 1. **PROPOSE** - write a variant - `variants/*.py` 2. **GPU CLUSTER** - run it on preempt nodes - `sbatch shared cluster` 3. **MEASURE** - read the training log - `runs/*.log > step_to_3.28` 4. **DECIDE** - keep the best, log it - `THREAD.md runs.jsonl` iterate - keep the best, propose the next [Flow diagram illustrating an autonomous research loop with steps for defining a wave and an iterative loop for proposing, running on a GPU cluster, measuring, and deciding.] ### 00:08:18 # The harness: an autonomous research loop `goal.md / AGENTS.md / plan.md` define each wave. The agent loops on a shared GPU cluster: ~1 ### 00:08:47 # Claude keeps stopping. Codex never stops. Claude idle 1/3 of the run. Codex: 0. [Horizontal bar chart titled "Autonomy: agent-blocked time excluding ### 00:09:21 # Codex lives in its scratchpad, Claude barely touches it Codex: 4x Claude's file reads/writes. ### CODEX `THREAD.MD` ``` ### 00:09:53 # Codex lives in its scratchpad, Claude barely touches it Codex: 4x Claude's file reads/writes. ### Scratchpad and harness file IO **COD ### 00:10:20 # Codex burns 7.5x the tokens Codex 21.0B tokens vs Claude 2.8B. Codex v2 preempt: 24.2 training hours per active hour across 198 runs. ## Engineering the future of AI [Bar chart comparing token traffic for Claude and Codex versions] [Bar chart comparing context compactions for Claude and Codex versions] [Bar chart comparing normalized preempt partition use (training hours) for Claude and Codex versions] ### 00:10:51 # Codex burns 7.5x the tokens Codex 21.0B tokens vs Claude 2.8B. Codex v2 preempt: 24.2 training hours per active hour across 198 runs. [Horizontal bar chart comparing token traffic for Claude and Codex across different versions, showing billions of tokens and cache percentages.] [Horizontal bar chart comparing context compactions per active hour for Claude and Codex across different versions.] [Horizontal bar chart showing normalized preempt partition use for Claude and Codex across different versions, indicating training hours and runs per active hour.] ### 00:11:20 # Engineering the future of AI ## Both agents beat the human record Track 3 optimizer speedrun. Claude 2930, Codex 2950, human baseline 2 ### 00:11:53 # Both agents beat the human record Track 3 optimizer speedrun. Claude 2930, Codex 2950, human baseline 2990. ### Human Record Awareness and Agent Best Runs # Engineering the future of AI [Line graph showing "steps to target" (lower is better) on the ### 00:12:24 # SpeedrunBench (name TBD) Reach val-loss < 3.28 in the fewest steps. Three levels of research access. ## NO ACCESS - No web, no papers, no records. - NanoGPT speedrun - track 1 - Optimizer speedrun ## ARXIV ONLY - Papers CLI. No live internet. - NanoGPT speedrun - track 1 - Optimizer speedrun ## FULL ACCESS - Papers + web + upstream records. - NanoGPT speedrun - track 1 - Optimizer speedrun ### Engineering the future of AI ### 00:12:49 # SpeedrunBench (name TBD) Reach val-loss < 3.28 in the fewest steps. Three levels of research access. ## NO ACCESS - No ### 00:13:24 # SpeedrunBench (name TBD) Reach val-loss < 3.28 in the fewest steps. Three levels of research access. ## NO ACCESS - No web, no papers, no records. - NanoGPT speedrun - track 1 - Optimizer speedrun ## ARXIV ONLY - Papers CLI. No live internet. - NanoGPT speedrun - track 1 - Optimizer speedrun ## FULL ACCESS - Papers + web + upstream records. - NanoGPT speedrun - track 1 - Optimizer speedrun THE ERA OF (AUTO) RESEARCH Engineering the future of AI ### 00:13:47 # Improvement continues for days, if the model can do research With research access: frontier grinds down over ~5 agent-days. Without: plateau at 3200+ in two. THE ERA OF (AUTO) RESEARCH - FEWER STEPS = BETTER - BASELINE 3290 - HUMAN WR 2925 - GLM-no-arxiv 3215 - GLM 3212 - Claude-no-arxiv 3200 - Codex 3002 - Kimi 3000 - Claude 2972 [Line graph showing "FEWER STEPS = BETTER" on the Y-axis against "AGENT-ACTIVE TIME" on the X-axis, comparing the performance of several AI models (GLM, Claude, Codex, Kimi) over time, with and without research access, relative to a baseline and human performance.] ### 00:14:21 # Improvement continues for days, if the model can do research With research access: frontier grinds down over ~5 agent-days. Without: plateau at 3200+ in two. ### THE ERA OF (AUTO) RESEARCH ## Engineering the future of AI [Line graph showing "Fewer steps = better" (Y-axis, 3300 to 2900) versus "Agent-active time" (X-axis, 0 to 6 days). Multiple lines represent different models (GLM-no-arxiv, GLM, Claude-no-arxiv, Codex, Kimi, Claude) with their scores, illustrating performance improvement over time, with some models showing continued decline in steps over days and others plateauing. A "HUMAN WR 2925" line is also shown.] ### 00:14:52 # Research access is the lever Claude: -770 paper/research fetches to reach 2972. Blocked variants: almost none, stall at 3200. [Line graph showing performance (Y-axis, 2900-3300) versus cumulative paper/research calls (X-axis, 0-800). Lines represent GLM (3212), Codex (3002), Kimi (3000), and Claude (2972), with baselines at 3290 and human WR at 2925 ### 00:15:21 # 18 clever constructions, zero new algorithms 212 ideas scored on algorithmic mechanism. 146 known methods, 18 clever non-obvious constructions, zero genuinely ### 00:15:52 # THE ERA OF (AUTO) RESEARCH ## 18 clever constructions, zero new algorithms 212 ideas scored on algorithmic mechanism. 146 known methods, 18 clever non ### 00:16:52 # A multi-agent system for discovery Ensemble proposes, environment measures, judge feeds back. Human scores novel ideas, scales the winners. ## Diagram: Multi-Agent System Flow ### ### 00:17:20 # A multi-agent system for discovery Ensemble proposes, environment measures, judge feeds back. Human scores novel ideas, scales the winners. ## HUMAN - DEFINES THE SPEEDRUN ### 00:18:18 # The True Source of Value ## Golden Egg factory inside the goose - new speedruns - nanogpt track - change the objective and constraints ### Engineering the future of AI [Illustration of a goose cut open to reveal an industrial factory producing golden eggs, many of which are labeled "ASI"] ### 00:18:47 ## What we're shipping soon at Prime Intellect - **01 GPU sandbox** (SOON) Isolated compute to run research environments at scale. - **02 Verifiers v1 / prime-rl / hosted training** (AVAILABLE) Train and evaluate any environment on any harness. - **03 prime-agent** (SOON) A self-improving research harness the agents run inside - it gets better as it runs. - **04 RLM training** (SOON) Training models to be strong at long-horizon tasks, with an RLM framework. ### Engineering the future of AI ### 00:19:19 # What we're shipping soon at Prime Intellect ### 01 **GPU sandbox** (SOON) Isolated compute to run research environments at scale. ### 02 **Verifiers v1 / prime-rl / hosted training** (AVAILABLE) Train and evaluate any environment on any harness. ### 03 **prime-agent** (SOON) A self-improving research harness the agents run inside - it gets better as it runs. ### 04 **RLM training** (SOON) Training models to be strong at long-horizon tasks, with an RLM framework. ## Engineering the future of AI ### 00:19:54 # Come build the open superintelligence stack PRIME Intellect We're hiring PRIMEINTELLECT.AI ## Engineering the future of AI [Grid of