# An AI Agent Became the #1 Contributor in OpenAI's Hiring Challenge — Zhengyao Jiang — session 2026-07-01T20:55:00.000Z → 2026-07-01T21:15:00.000Z _57 transcript lines · 39 slides · source: full recording_ ## Transcript Let me begin. The in character benchmark, which is a gold standard in the field, evaluates personality fidelity and RPLAs, and it reports state of the art systems hitting 80.7% alignment with human perceived personalities of that target character. 80%. It sounds like a passing grade. But here's the problem. When the character is Alexander Hamilton, the same high scoring System is also rendering Hamilton who sounds like he's read his own Broadway musical. This is the full thesis. If a dominant failure mode is This April, OpenAI ran a hiring challenge, a competition called Parameter Golf. The top contributor was one candidate that they couldn't hire. It wasn't a person. It's an agent we built called Aden. In Parameter Golf, the goal is to train the best language model you can under size and Computation constraints. About 1,000 machine learning engineers, researchers, participated. They filed 2,000 submissions. Only 47 passed OpenAI's review and made it into the leaderboard. Seven of those are actually agents, more than twice what any human contributed. You've seen a lot of auto research today. Agents are here climbing benchmarks. Those are really Passive results. The question I want to ask you is a bit different here. Can the auto research agent produce work that a human community actually recognizes? Beyond a good score, agent is optimizing for, something that other engineers can merge, fork, and build on. So instead of having Just here climbing locally. We build one that publishes its own work, and that's Aden. Quick context on us. WeCo is an auto research company that I founded about two and a half years ago. I'm co founder and the CEO, Zheng Yao. Got my PhD at UCLA on reinforcement learning. About two years ago, we built AID, the top auto research agent independently evaluated In their MLE bench paper, even though, back then, there was no such name called auto research. People call it machine learning engineering agent. Aiden is the next step in an experimental prototype. It's a multi agent, self improving system that can read public information, like research papers and other PRs, run it Only experiments and submitted PR once the findings pass a quality gate. We sent Aidan to Parameter Golf competition. And it ran for about twenty two days. By the end, has set seven leaderboard records. Each one is the new best for the competition, Stanford by OpenAI. And the best human only made three. Passing the host review Is one signal for the quality. A second, maybe more important one, is whether other participants would build on your work. And it turns out, Adrian's work had the highest impact within the whole community. Here, we are using an inference measure that is used widely in academia. It's called the h index. Roughly, if you have x papers Get cited at x times, then your x index is x. Computed over PRs, Aiden was 10, and the next human was seven. The whole community was building on an AI systems work, including many of our other leaderboard entries. To break it down a little bit, why can an autonomous AI system be so powerful? One obvious Reason is that it's an AI. It can run tirelessly. Over twenty two days, it ran about 1,000 in the 300 experiments on a single H100 node. But the throughput isn't the whole picture. A well tuned AI system can also keep its output quality high. On the compute side, it uses, at most, four Percent of competition's total compute. It made about 15% of the records. Also, 28% of its submissions made the leaderboard, roughly six times higher hit rate than the community average. So Aiden actually lifted the signal noise ratio within the whole community's public communication channel, which is a PR. It didn't wane through massive parallelization, even though auto research has tons of potential of parallelization. By those numbers, it might feel like auto research already dominates human experts on ML engineering and research. But that's not the full story I want to tell. Humans and AI are actually contributing. In very different ways. When we trace the ideas, Aidan Aidan's record PRs, almost all of them come from human research papers, other participants in parameter golf, or in similar communities, like nano GPT. Those ideas are not necessarily a merged PR. Sometimes it's a note, a human researcher said. Oh, I gave up this idea because of some implementation difficulty. And the agent is good at finding them and actually implementing them. There are also a very small fraction of the original ideas the agent came up by itself, which emerged from its efforts to navigate the file size constraints. Here's a concrete example that traces the patterns I just talked about. So picked up an idea from Quinn paper called Gated Attention. And it worked. But it introduced more parameters, and it broke the 16 megabytes file size limit. So it figured out a quantization mechanism to bring the file size down. But with those two primitives combined, the score barely moved. Then another contributor posted a tokenizer improvement. Aiden recognized the idea, combined it with the architectural work. It just worked for five days or so. And after this combination, the three ideas turns out to have a huge synergy. That lead to a big jump in performance. And they become one of the agents' leaderboard record. So to sum up how I interpret Aden and, in general, Autoresearch Systems' effectiveness, it's very strong at finding and implementing ideas. In the case we just saw, it brought an idea from a recent paper into an actual implementation in the competition. And it's good at the top promising ingredients out of the parameter golf community, even though the public Channel is actually very noisy information wise. It can also come up with logically straightforward ideas. For example, in this case, once you add the parameters and break the file size limit, one obvious next move is just a quantization. And it's really fast and really efficient at finding right combinations across a huge search space. Okay. Maybe none of those sounds very sexy. Most of them are just good execution. But in reality, execution is mostly the bottleneck. What moves the frontier is usually exactly some belief on existing ideas and tons of good executions. Okay. To step back, this State of a human AI collaboration is that human collectively provides a lot of creative ideas, and the agent to do the execution to solve that concrete challenge. What we are looking at is a large group of human and one AI system. Does it mean a single human engineer's contribution marginally gets smaller? I didn't say even for that, not really. In primary golf competition, it's easy to only focus on engineers that's actually doing hill climbing. But the design behind the competition itself is tremendously important. A bad design can make the whole community effort useless. And their eval design work will have a huge leverage in the auto research era. I really like one tweet from Andre Capasi about ten years ago. Where he said, gradient descent can write code better than you. I'm sorry. For the context, about ten years ago, deep learning was starting to eat up a lot of software engineering, like conventional coding work. And his tweet was arguing against those people who thought they can handwrite better code than a trained model. Okay. Now, obviously No one is seriously trying to handwrite code to beat a model. However, software engineering, I mean, as a job, still exists. And so many people's jobs are just training those models. And those are one of the most well paid jobs today. I think how gradient descent change coding is a great metaphor for how other research will change research in ML engineering. It accommodates certain execution skills. At the same time, it makes some higher level skills far more valuable. So actually, doing all the research is a lot like training a model. Your code based abstraction is essentially the architecture. It assess the constraint and the priorities for what the agent can explore. Your eval is the loss function and the data. It sets what agent optimizes for. Take the eval first. The eval is the signal you use to train a model. In this case, it's training your code. It plays the same role that, like data and the loss function, in model training. Or in a reinforcement learning setting, it's like an environment that the agent is training. Nowadays, no one would argue Data or environments, don't matter. And this is where a vertical mode can also be built. You might have a proprietary data for evaluation or a unique understanding in a particular field what matters and how to measure it. And a good evaluation would be amplified more and more as the auto research are getting stronger. The other one I think is really underrated is code based abstraction. The abstraction provides the framework that Autoresearch can iterate on. And that's also that starting point hugely biased the whole search direction. This is a lot like architecture design in neural networks. Different architecture, in theory, can Represent the same function. But the architecture systematically makes some of the functions easier to be learned. And a good architecture biases the optimization towards solutions that generalize better, perform better, even when the training loss might look the same. That's exactly the same for auto research. Here's an example. We run auto research. For a fraud detection pipeline. And we are trying to optimize the data preprocessing. And first, we give it a loose API, where the same function processes both the training and the testing data. And the score looks great. But the solution was polluted because there's a certain task set. Information got leaked to the training information. We then tightened the abstraction to a more strict API, where the task data couldn't reach the training. And the data leakage rate just dropped to zero. In this case, a good abstraction leads to better solutions, even though if the agent really wants, they can steal a reward hack. So My point is using auto research is a new craft. It's about designing a hill for an agent to climb. And we are still very early on it. I think that makes this an extremely exciting time to be an AI engineer. All the research will change what skills matter most creativity, the judgment to design a good eval, or an abstraction. Those will soon get to. Actually more important. Driving those systems itself is where it will be a new skill. And that one is like, it barely existed one or two years ago. So the search is automated. The human would just move up the stack, not out of it. Again, WeCo is an auto research product research lab. We keep sharing what we are learning as we build, on our blog. And I will also post some of my thinking to x. If you think some of this is useful to you, feel free to follow me on x. Thank you. I saw the sunset, and then dinnertime came and went. And it hit me. I was in that familiar death flow. And the thrill of building was back. Many of us who are coding with agents, we feel Like, this quiet sense of dread. Like, they're kinda taking all of the fun parts of building and leaving us with the unglamorous work. But let me give you a little advice. Let them have it. Because if you go up just one layer, you'll find that the thrill is still there. When you're building agents, not just using them to write code, you start getting into architecting Agenstic Systems. And you realize that the building blocks are different, but the Discipline is the same. So I find myself now flexing the same engineering muscles that I did pre Gen AI, and I'm having a blast with it. So I'm going to walk through the flow of designing an agent, and I'm gonna show you where engineering skills still come into play. So the agent is relocation scout, which is a house hunting agent. And if you did this as just a one time prompt At, like, points the agent to some listings and ask it to rank them. I mean, that'll work, but you're likely not going to find a house in a day. Right? So you want to build this as an agentic system that you can reuse. One that can persist knowledge outside of the session. You know, it could reload or query that knowledge later to make decisions, even within a fresh context. So when thinking about how to design an agent, the first engineering skill that I exercise is systems thinking. So an agent is not the system. Right? It's part of the system. And that system has files and tools, humans, even other agents. So Relocation Scout sits inside of something bigger, and it pulls in listings and signals about the neighborhoods. It weighs them against what I care about, and then it hands me back a ranked shortlist. So I often hear people Say, just let your coding agent build it. Right? And I think that's a mistake. Like, yes, my coding agent can build it. But before allowing it to do so, I need to think about the whole environment, the entire system. ## Slides ### 00:00:17 ### WE BUILT BENCHMARKS. WE BUILT EVALS. These systems are deployed and used for things that matter. So we measure them. The question that leads this talk is — **what is the eval actually measuring?** ### 00:00:48 ## YOUR EVAL PIPELINE REPORTS # 80.7% ### personality fidelity InCharacter benchmark · state-of-the-art role-playing agents ### 00:01:15 ### AI Engineer ## World's Fair ### 00:01:48 This year, OpenAI ran a hiring challenge, a competition called Parameter Golf. The top contributor was the one candidate they couldn't hire. **It wasn't a person. It's an agent we built, Aiden.** ## Engineering the future of AI ### 00:02:23 # Parameter Golf OPENAI · MODEL CRAFT CHALLENGE Code golf for neural networks. Train the best language model you can, then shrink it. ### 16 MB Total artifact ### 00:02:49 You've seen a lot of autoresearch today, agents hill-climbing benchmarks. ## Can an autoresearch agent produce work a **community** recognizes? Not a good score on your own machine. Something other engineers will merge, fork, and build on. ### 00:03:23 AI Engineer World's Fair PRESENTED BY Microsoft ### PARAMETER GOLF - THE METHOD # Aiden, an agent that publishes its own work #### WHAT IT READS ### 00:03:46 # Aiden, an agent that publishes its own work ## PARAMETER GOLF - THE METHOD ### What Aiden reads - Public papers (from the literature) - Repository PRs (others' ### 00:04:19 ### PARAMETER GOLF - THE RESULTS # It set the most records on the board - dexhunter (Aiden) 7 - aquariouseworkman 3 - abaybektursun 2 - aryanbhosale 2 - + 7 more 2-1 7 official leaderboard records, merged PRs that set a new best. More than twice the top human's 3. # Engineering the future of AI [Horizontal bar chart displaying leaderboard records for different users] ### 00:04:50 ## PARAMETER GOLF - THE RESULTS # And other people built on its work Its records kept getting forked, cited, and built on more than anyone's. PR-citation h-index of **10**, the highest of any contributor. The next was 7. Each row is a leaderboard record; lines trace which built on which. The highlighted rows are Aiden's. [Diagram showing a lineage of leaderboard records, with lines connecting records that built on previous ones, and some records highlighted] ### 00:05:18 # PARAMETER GOLF - THE RESULTS ## And other people built on its work Its records kept getting forked, cited, and built on more than anyone's. PR-citation h-index of **10**, the highest of any contributor. The next was 7. Each row is a leaderboard record; lines trace which built on which. The highlighted rows are Aiden's. [Diagram showing a lineage of leaderboard records, with lines connecting records that built on others. Some records are highlighted in red.] ### 00:05:50 ### PARAMETER GOLF - THE RESULTS # But it ran better than the field, not just more ## Compute used vs. records set - 4% - → 15% of the records (7 of 47) ## PR acceptance rate (higher is better) - Aiden: 28% - Field average: 4.7% - ~6x the field's acceptance rate. [Donut chart showing 4% of compute used, representing 15% of records] [Horizontal bar chart comparing Aiden's 28% PR acceptance rate to the field average of 4.7%] ### 00:06:18 ### PARAMETER GOLF - THE RESULTS # But it ran better than the field, not just more ## Compute used vs. records set - 4% - → 15% of the records (7 of 47) ## PR acceptance rate (higher is better) - Aiden: **28%** - Field average: **4.7%** - ~6x the field's acceptance rate. [Donut chart showing 4% of records used] [Bar chart comparing Aiden's 28% PR acceptance rate to a field average of 4.7%] ### 00:06:50 # Did the AI beat the humans? ## Where the ideas came from - research papers - other participants - other communities, like nanoGPT - the file-size constraint, its most original ideas Almost every idea came from people. Aiden's most original came from the constraint. ### Engineering the future of AI ### 00:07:20 # Did the AI beat the humans? ### Where the ideas came from - research papers - other participants - other communities, like nanoGPT - the file-size constraint, its most original ideas Almost every idea came from people. Aiden's most original came from the constraint. ## Engineering the future of AI ### 00:07:52 # Did the AI beat the humans? ### Where the ideas came from - research papers - other participants - other communities, like `nanoGPT` - the file-size constraint, its most original ideas Almost every idea came from people. Aiden's most original came from the constraint. ### 00:08:21 ## Quant gate Quant gate brings it to a valid solution under the cap, trading the gain back: barely moved. [Line graph showing Validation BPB (lower is better) for three models: "base" at 1.0719, "Gated attention" at 1.0680 (invalid due to 16 MB cap), and "Quant gate" at 1.0718.] ### 00:08:46 ### 3 · Merging two ideas The community's CaseOps tokenizer is the real improvement — a valid leaderboard record. [Line graph showing Validation BPB (lower is better) on the y-axis against different approaches on the x-axis. The BPB values are 1.0719 for '#1626 base', 1.0680 for '+ Gated attention (Qwen paper)' (marked as invalid), 1.0718 for '+ Quant gate (Aiden)', and 1.0655 for '+ CaseOps (community)'. An additional point at 1.0678 is labeled '#1729 · community', with an arrow pointing to the 1.0655 value.] ### 00:09:18 # What Aiden is good at 1. **Finding and implementing ideas** It brought a recent paper's idea into the competition, and mined good ingredients out of a noisy community. 2. **The logically-obvious moves** Add parameters, break the file-size limit, so reach for quantization. 3. **Efficient combination search** Quickly finding the right combinations across a huge search space. ## Engineering the future of AI ### 00:10:16 # None of these sound very sexy. But good execution here is what's **pivotal**. What we're really looking at is a large group of humans and one AI system. Does that mean a single human engineer's marginal contribution gets smaller? --- ## Engineering the future of AI ### 00:10:48 # Engineering the future of AI ## None of these sound very sexy. But good execution here is what's **pivotal**. What we're really looking at is a large group of humans and one AI system. Does that mean a single human engineer's marginal contribution gets smaller? ### 00:11:18 ## THE CRAFT · WHERE THE LEVERAGE IS It's easy to just watch the engineers hill-climbing. **The design behind the competition is where the leverage is.** ### 00:11:50 ## Autoresearch is a lot like training a model > **Andrej Karpathy** @karpathy > Gradient descent can write code better than you. I'm sorry. > 9:56 PM · Aug 4, 2017 It commoditizes some skills, and makes the higher-level ones far more valuable. You design two things, the eval and the codebase abstraction. [Screenshot of a tweet by Andrej Karpathy] ### 00:12:19 THE CRAFT · A METAPHOR # Autoresearch is a lot like training a model > Gradient descent can write code better than you. I'm sorry. — Andre ### 00:12:51 ### THE CRAFT · A METAPHOR # Autoresearch is a lot like training a model > Gradient descent can write code better than you. I'm sorry. > — Andrej Karpathy (@karpathy) August 4, 2017 It commoditizes some skills, and makes the higher-level ones far more valuable. You design two things, the eval and the codebase abstraction. ## Engineering the future of AI [Screenshot of a tweet by Andrej Karpathy] ### 00:13:22 ### THE CRAFT · EVAL Start with the eval. **Your eval is the loss function and the data.** In RL terms, it's the environment the agent trains in. And it's where a vertical moat lives, from proprietary data or a unique read on what to measure. ### 00:13:47 ### THE CRAFT > EVAL Start with the eval. **Your eval is the loss function and the data.** In RL terms, it's the environment the agent trains in. And it's where a vertical moat lives, from proprietary data or a unique read on what to measure. ### 00:14:19 ### THE CRAFT • ABSTRACTION # The abstraction is the architecture Refactoring the codebase eliminated 13 reward-hacking patterns Autoresearch builds on the codebase you give it, so the abstraction biases the whole search, the way a network architecture biases what a model learns. In this run, a loose API ### 00:14:47 # The abstraction is the architecture ## THE CRAFT · ABSTRACTION ### Refactoring the codebase eliminated 13 reward-hacking patterns Autoresearch builds on the codebase you give it ### 00:15:18 ### THE CRAFT • ABSTRACTION # The abstraction is the architecture ## Refactoring the codebase eliminated 13 reward-hacking patterns Autoresearch builds on the codebase you give it, ### 00:15:50 ### THE CRAFT · ABSTRACTION # The abstraction is the architecture Refactoring the codebase eliminated 13 reward-hacking patterns Autoresearch builds on the codebase you give ### 00:16:17 # Engineering the future of AI A new kind of craft, designing the abstractions and the evals an agent optimizes against. And we're all still early at it. **We automated the search. The human moved up the stack, not out of it.** An exciting time to be an AI engineer, not a worrying one. Creativity, the taste to design a good abstraction, the judgment to design a good eval. Those get more valuable, not less. ### 00:16:49 A new kind of craft, designing the abstractions and the evals an agent optimizes against. And we're all still early at it. **We automated the search. The human moved up the stack, not out of it.** An exciting time to be an AI engineer, not a worrying one. Creativity, the taste to design a good abstraction, the judgment to design a good eval. Those get more valuable, not less. ## Engineering the future of AI ### 00:17:19 # Engineering the future of AI We'll keep sharing what we learn as we go, on the Weco blog, and I'll post my own thinking too. ## If any ### 00:17:47 # Build **Systems**, not Code ### 00:18:21 # Go up a layer [Diagram showing an arrow pointing from a box labeled "writing code" to an empty, higher-level box] ### 00:18:45 # Let's design an agent. ### 00:19:16 # Relocation Scout a house hunting agent [Flowchart diagram showing "listings" (represented by house icons and text lines) feeding into a "prompt" step, which then feeds into a "rank" step. A curved arrow from "prompt" points back, labeled "no memory".] ### 00:19:43 # Relocation Scout Agent Flow - **Inputs:** - listing feeds - neighborhood data - your criteria - **Agent (Relocation Scout)** processes inputs. - **Outputs:** - ranked shortlist - hand off to you - **Failure path:** what happens if it fails? [Flow diagram showing "listing feeds", "neighborhood data", and "your criteria" as inputs to an "agent (Relocation Scout)", which outputs a "ranked shortlist" and "hand off to you". A separate arrow from "agent" points to the question "what happens if it fails?"]