# Applied Research, Prime Intellect — Will Brown — session 2026-10-14T19:45:00.000Z → 2026-10-14T20:03:00.000Z _30 transcript lines · 52 slides · source: full recording_ ## Transcript **[00:00:54 · 0]** I'll do. Good to go. Hey, how's it going? I'm Will Brown. I lead applied research at Prime Intellect, and hope you're all having a great AI Engineer conference. **[00:01:11 · 0]** This is always one of my favorites. It's always just so cool to see what everyone's working on and building. And I feel like it is a place where people get to talk about what the future looks like and start seeing the trends in the industry and seeing the progress and what works and what doesn't. And that's kind of the theme I want to think about for this talk today is, like, what comes next from what we're doing right now, and what are some of the things that we've been working on and also just been seeing and research that's unfolding that kind of points at what we think the future of AI might look like at the organization level? Because we've seen a lot of very cool progress in agents and interfaces and bots and things like that over the past several months, years. **[00:02:00 · 0]** But there's always, like, there's always more coming. Things are just moving so fast. And I think it's a very exciting time to just imagine and speculate and plan for what comes next. And so I don't know if any of you remember, back in the early ChatGPT days, OpenAI had kind of spelled out their framing of, like, what are the different stages of AI progress on the road to AGI? And I think we moved along this faster than we thought we would. **[00:02:33 · 0]** We, like I remember eighteen months ago, I was here in New York at I Engineer giving a talk where we had gotten reasoners, and chat bots had been around for a while, and reasoners were like the new thing. But we didn't really have agents yet. And now we've climbed pretty quickly through agents and innovators. And the question next seems to be, Okay, what happens for organizations? Like, what does it mean for an organization to truly be AI native, not just as a startup that uses a lot of agents and builds everything with cloud code, but at the large enterprise scale? **[00:03:12 · 0]** What do companies of the future look like that involve the deep integration of agents throughout the lifecycle of the organization, and what do we need to build on the path there. So a bit about Prime and Elect, just for some backdrop. We build a lot of different things around AI infrastructure for training, deploying, continuously improving models, and especially open source models and open source tooling. And so we do a lot of things, and we roughly do the sorts of things that you would do inside of a big lab. And these are our products. **[00:03:48 · 0]** And the reason we do this is because we are imagining a future that is coming pretty quickly, it seems, where open source models are readily available to train and deploy on your own workflows, on your own data, in your own harnesses, and can be the sorts of things that organizations can have grow with them over time that allow compounding their institutional knowledge, their institutional advantage from their workflows, from their data, from their people and their processes, in addition to using the models from the Frontier Labs. And so this talk will be a bit about how we see that evolving and unfolding, where, like, what is the role of training in an AI native enterprise? How does this coexist with the Frontier models? And what do we think is going to be the version of this that really unlocks the most value for people, and what does it look like? And so organizations have lots of knowledge. **[00:04:39 · 0]** And the longer an organization's been around and the kind of more established it is, the more implicit knowledge there is that kind of doesn't ever get written down. There are things that do get written down. There's code bases. There's emails. There's processes. **[00:04:53 · 0]** Some of these are very explicit. They're like rules of, like, you must do this. This thing works like that. But a lot of things kind of live in people's heads, and they live in between the lines of emails, of the meeting notes, of the kind of who talks to who. A lot of companies will have an on paper org chart and then a real org chart of who actually makes decisions and how does change get adopted throughout the company, what source of mandates are likely to take effect. **[00:05:19 · 0]** And people have kind of built up this knowledge from being at companies for a long time of, like, oh, this is kind of how things work. This is where the friction points are. And I think in many cases, and I think this is really one of the big opportunities for AI at the organizational level is, is that many individual people are kind of not in a position where they can change the things that are broken in an organization, because no one is actually spelling it out. No one is taking a bird's eye view of looking at everything that happens in an organization and saying, hey, this over here is inefficient. This could be better. **[00:05:51 · 0]** This thing keeps happening across all these different teams, but nobody's figured out that the solution has actually been discovered by that team. And these are the sorts of things that agents are great for. Agents are very capable at combing through large amounts of textual data that's unstructured and finding these kinds of patterns and turning them into something useful and surfacing them as, like, Okay, this workflow has a problem because it is wasting time, and it's inefficient, and it could be different instead. And we don't want to just do this once, though. We want this to kind of become the way that organizations adopt AI broadly, where agents are able to get the context they need, but also they're able to improve both the agents themselves as well as the organizations. **[00:06:35 · 0]** And we want agents to be more like our coworkers. But human coworkers learn on the job. Agents currently learn on the job in relatively primitive ways, and they do this via in context learning. And oops, sorry. I'm going to go back a few seconds so we can edit this on the recording. **[00:07:01 · 0]** Agents learn in relatively primitive ways through in context learning. And this typically looks like them kind of having a skill file written by somebody or looking around a codebase, but they don't get the institutional level latent knowledge that human employees really do. And so they relearn a lot. They have to rediscover things. They spend a lot of input tokens. **[00:07:22 · 0]** And I think nowadays people are realizing that cached input is the bottleneck in many agent costs. And even if you do get great cache rates and cache costs go down, you still have to reprocess a lot of input to redo a workflow every single time. And so to get around this, you kind of want to be able to bake in more into the model, especially for things that are harder to write down as rules. So I like thinking about the spectrum from rules to patterns, as well as conceptually code versus data. And so lots of things live as code, and skills are kind of like this, rules are kind of like this. **[00:07:56 · 0]** But things like preferences and friction, heuristics and trends, these are all data. And institutions have so much data that they are running on, and the employees of the organization implicitly learn that models never really get a chance to learn. And this is the sort of thing that we see as blocking them from becoming true coworkers. And so there is a question of, like, Okay, what goes in context? What goes in the weights? **[00:08:21 · 0]** And part of it is you kind of need to earn it. Like, things don't just go in the weights right away. But if you see kind of every new generation of models, the models get better not because the prompt is better, but because they train them more. Like, training is useful. Training works. **[00:08:36 · 0]** Training is why we see the progress we do. And the model is roughly, like, trained on lots of data that is aiming to approximate things that the labs expect will be high value, repeatable workflows at organizations. And I like this I think there's a process that currently works. It's how the models get better. But if you spell it out, it's relatively inefficient. **[00:08:58 · 0]** So the way that a model would learn how to get really good at using some combination of Salesforce tools, for example, let's say, is someone who knows how those tools get used has to be looking for work. And maybe they previously worked at a place that were using these tools, and now they are browsing expert network sites to make some money in their free time. And they are sharing their knowledge to someone who works at an expert network company. This person at this network company will then take this data from the human and pass it to someone else whose job is to create RL environments. This RL environment might get sold then by someone who doesn't have this expert knowledge to a researcher at a lab who also really isn't an expert in the domain, who then will be refining it to put it into a training pipeline, to go into the next frontier model run, to make a model that has, kind of through this game of telephone, gotten some knowledge of that human originally. **[00:09:56 · 0]** But there's so many steps in this pipeline, and it's a relatively secret pipeline. Like, people talk across two stages of it, but the human who knows this thing is not supervising the training run, is not really directly inputting how the model gets trained in the way that a human could if they were doing it for their coding agent that they're working with day to day. And so what we see as an opportunity, and kind of also where we would hope to go, is to close this gap and to allow the person who the same way that when we work with our coding agents, we're prompting them the same things every time, telling them to not repeat the same mistakes. This is how you could train models. You can tell the models things, and they will learn these things durably, not just as a line in your agents MD, which kind of accumulates, and then you have to refactor it all the time. **[00:10:42 · 0]** You can do some of that. But a lot of times, what you really want is you're giving kind of contextual preference. You're saying, in this setting, this thing that you did was not exactly what I wanted, and I would have preferred this instead. And we'd want systems that can compound this into durable learning. And training is a very effective way of doing this, because the shape of data you're giving there is quite close to what you would want from an actual training pipeline. **[00:11:09 · 0]** And so there's several different types of learning. There's things that people do. There's things that you can do at the harness layer with skills and memory. There's knowledge that just lives throughout an organization in file systems and databases that you retrieve with tools and with code. Then there's the models themselves, which have this really broad world knowledge. **[00:11:28 · 0]** But we all have, I'm sure, experienced frustration when using a model whose, like, cutoff of knowledge is, like, a while back, and, like, you can't really talk to the models about the news of the past six months nowadays. Like, they don't really they have to go, like, relearn a world model in context every single time, and this just does not allow them to really operate as fluently as they can discuss things that happened longer ago. And so organizations, to get the most value out of models, we imagine they will want to increasingly have models that really understand what is going on in the organization at an intuitive level. And this is the sort of thing that you want to train for. And so how do you get there? **[00:12:08 · 0]** Like, what is the road for an organization to go from having this idea in the abstract of, like, we're going to train models on our data to actually getting there? And I think one really useful starting point is evals as essentially test cases. I think with code, we've gotten to a point of maturity around testing and life cycles and software development life cycles for ensuring that code bases grow healthily over time, usually. And agents are doing so much of this work for us. And the version of continual learning that I think is most practical for most organizations is just treating a model kind of like a code base, where it is the sort of thing that you can have a lifecycle around, you can add test cases to, to, you can add features to, you can add assets to. **[00:12:55 · 0]** And this is the sort of thing that the process of using agents like, initially, you collect traces. These traces have your prompts, and they have your feedback to things that models did. And this, just from the process of using these models, you are accumulating the assets you need to create evals. It's just that making evals historically has been a little bit painful, and a lot of organizations have not necessarily wanted to do it manually. But the agents are quite quickly getting to the point where, thankfully, we have robots that can do a lot of the evalcuration for us, especially as we train them more on the best practices of making evals in ways that are robust and amenable to learning. **[00:13:35 · 0]** And so I think another useful example here is the decision model craze that has kind of been taking the world by storm in the past couple weeks with Jev, and also the parallels around routing and people being very excited about cost optimization because we see that it's possible to spend very large amounts of money on cloud every month, and we figure out, like, when should we use a cheaper model? And this is the sort of thing that I think people are recognizing needs to kind of live in the harness. But it also is an attribute that looks relatively similar to decision models. And the reason I bring both these up together is because there's things with very clear labels where there's a right answer, and there's an opportunity for a human or another agent to give feedback. You can have the big model and the small model do the same task, judge whether the Small Model was as good as the Big Model. **[00:14:20 · 0]** If so, then this is now data of, like, for this kind of thing, we can start routing to the Small Model more. Same for a decision model that's learning to kind of mimic the way that a human in a certain situation would make a judgment. This is a great labelable action, where the human might be doing the approval the same way that we approve our tool calls in Cloud Code sometimes. But I think most people have turned on auto tool acceptance. But every organization for more bespoke things that aren't just about tool security are going to have their own rules and preferences, and these are the sorts of things that you could then compound into a decision model. **[00:14:54 · 0]** And so this is a very believable form of continual learning that a lot of people could just plug in tomorrow and see immediate value from. But more broadly, what we see people wanting to do and this is a lot of the work that we do at Prime Intellect is working with organizations that are looking to do relatively ambitious training projects around turning their traces from deploying a model in production for a high volume use case into rich, robust environments that are blended, real, and synthetic that kind of capture the complexity of the real tasks they want to be using long running agents for. And agents are very useful with sufficient orchestration around them, sufficient tooling, and humans in the loop at turning these into environments that are rich enough to train in. And we see that this is getting more and more automated with every new generation of open and closed models. And this means that we have an opportunity to start having this whole thing of taking the stuff that we're already doing and turning it into a self improvement loop just kind of fall out magically from applying agents to curation of the data and creating interfaces for humans to give signal about what they want. **[00:16:03 · 0]** This is an idea that I've been, for my own setup, been playing around with a lot that I think will become a popular interface for interacting with models more in the future. You're seeing it a little bit in some of the bots, but I think it especially now that models are good at writing code on the fly, these kind of widget card features that are, I think, a very practical form of dynamic UI that are also really useful for eliciting human decisions. So they're kind of like the tool call approval thing, but much more personalized, much more specific, and also kind of get around the problem of, like, Okay, you have 20 different bots that are going, and you have to remember which ones are where, and who is curating all this for you. And then people are like, do you want one bot? And then you're like, oh, one's great, but I actually want two. **[00:16:47 · 0]** Then you have two. Then you have 20. And I think we kind of are oscillating back and forth between the single stream and the multi stream agent threads. But what we really want to do are make decisions. We want to give ideas to the agents, let them run with the ideas, and then come back to us when we need to make decisions. **[00:17:03 · 0]** And the agents hopefully should make good choices about when to bring us decisions. And if agents can create interfaces on the fly for us that allow us to very directly give them the information they need, one, this kind of is much more ergonomic for us as humans than reading, trying to parse through a long thread message to see, like, Okay, where was the question? It's also kind of like the human dual side of the decision model, dev style. But it also is perfect data. And so it's a setting where we've been experimenting with this as a way of training models to kind of get this feedback loop going from having human input with these kinds of natural interfaces be the sort of thing that directly acts as a label and is the sort of thing that a model can directly improve on because there is a very explicit target. **[00:17:51 · 0]** And it gets around a lot of the problems that we kind of hear about in reward hacking and whatnot. Those things are worth addressing, too, but this is where we see a ## Slides ### 00:00:00–00:00:20 # AI Engineer ## NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night, featuring prominent buildings like One World Trade Center, the Empire State Building, and the Statue of Liberty] ### 00:00:20–00:00:40 # AI Engineer ## WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Illustration of the New York City skyline at night] ### 00:00:40–00:01:00 # AI Engineer ## NEW YORK ### 00:01:00–00:01:20 # self-improving organizations WILL BROWN • HEAD OF APPLIED RESEARCH, PRIME INTELLECT [Misty forest scene with a body of water and a tall, rectangular monolith in the distance] ### 00:01:20–00:01:40 # self-improving organizations ## Will Brown ### Head of Applied Research, PRIME Intellect [Dark, misty forest scene with a body of water in the foreground and a glowing, rectangular structure in the distance] ### 00:01:40–00:02:00 # self-improving organizations WILL BROWN • HEAD OF APPLIED RESEARCH, PRIME INTELLECT [Misty forest landscape with a body of water and a tall, rectangular, translucent monolith on a small island] ### 00:02:00–00:02:20 # self-improving organizations WILL BROWN • HEAD OF APPLIED RESEARCH, PRIME INTELLECT [A misty, forested landscape with a reflective, rectangular monolith in a body of water] ### 00:02:20–00:02:40 # from chatbots to organizations - chatbots - reasoners - agents - innovators - **organizations** OpenAI's levels of AI capability [Diagram showing a progression of AI capabilities from chatbots to organizations, with "organizations" highlighted] ### 00:02:40–00:03:00 # from chatbots to organizations - chatbots - reasoners - agents - innovators - organizations OpenAI's levels of AI capability [Diagram showing a progression of AI capabilities from chatbots to organizations, with "organizations" highlighted.] ### 00:03:00–00:03:20 # from chatbots to organizations - chatbots - reasoners - agents - innovators - organizations OpenAI's levels of AI capability [Diagram showing five ascending levels of AI capability: chatbots, reasoners, agents, innovators, and organizations, with organizations highlighted.] ### 00:03:20–00:03:40 # from chatbots to organizations - chatbots - reasoners - agents - innovators - **organizations** OpenAI's levels of AI capability [Diagram illustrating a progression of AI capabilities from chatbots to organizations, with "organizations" highlighted] ### 00:03:40–00:04:20 # the prime intellect stack ## PLATFORM - **training**: managed RL on your environments - **inference**: serve open models and the ones you train - **observability**: metrics, rollouts, logs, checkpoints - **evaluations**: compare models and agents on your tasks - **sandboxes**: isolated code and tool execution - **research partnerships**: environment design and model development ## OPEN SOURCE - **verifiers**: environments for evaluation and training - **prime-rl**: async RL training at scale - **Prime Agent**: a coding and research agent - **INTELLECT series**: open models and training data ### 00:04:20–00:04:40 # the prime intellect stack ## PLATFORM - **training**: managed RL on your environments - **evaluations**: compare models and agents on your tasks - **inference**: serve open models and the ones you train - **sandboxes**: isolated code and tool execution - **observability**: metrics, rollouts, logs, checkpoints - **research partnerships**: environment design and model development ## OPEN SOURCE - **verifiers**: environments for evaluation and training - **prime-rl**: async RL training at scale - **Prime Agent**: a coding and research agent - **INTELLECT series**: open models and training data ### 00:04:40–00:05:00 # what organizations know - **WHAT A COMPANY KNOWS**: how the work gets done - **WHERE IT LIVES**: procedures, checklists, habit - **WHAT A COMPANY KNOW ### 00:05:00–00:05:20 # what organizations know - **WHAT A COMPANY KNOWS**: **WHERE IT LIVES** - how the work gets done: procedures, checklists, habit - past decisions, and how they turned out: memos, drives, people's memory - its customers, contracts, and numbers: its systems - who decides, who to ask: the org chart, and the real one - **judgment, norms, what good looks like**: **people** ### 00:05:20–00:05:40 # what organizations know | WHAT A COMPANY KNOWS | WHERE IT LIVES | | :----------------------------------------- | :---------------------------------- | | how the work gets done | procedures, checklists, habit | | past decisions, and how they turned out | memos, drives, people's memory | | its customers, contracts, and numbers | its systems | | who decides, who to ask | the org chart, and the real one | | **judgment, norms, what good looks like** | **people** | ### 00:05:40–00:06:00 # what organizations know | WHAT A COMPANY KNOWS | WHERE IT LIVES | |---|---| | how the work gets done | procedures, checklists, habit | | past decisions, and how they turned out | memos, drives, people's memory | | its customers, contracts, and numbers | its systems | | who decides, who to ask | the org chart, and the real one | | judgment, norms, what good looks like | people | ### 00:06:00–00:06:20 # what organizations know | WHAT A COMPANY KNOWS | WHERE IT LIVES | | :---------------------------------------- | :---------------------------------- | | how the work gets done | procedures, checklists, habit | | past decisions, and how they turned out | memos, drives, people's memory | | its customers, contracts, and numbers | its systems | | who decides, who to ask | the org chart, and the real one | | judgment, norms, what good looks like | people ### 00:06:20–00:06:40 # what organizations know | WHAT A COMPANY KNOWS | WHERE IT LIVES | | :---------------------------------------- | :---------------------------------- | | how the work gets done | procedures, checklists, habit | | past decisions, and how they turned out | memos, drives, people's memory | | its customers, contracts, and numbers | its systems | | who decides, who to ask | the org chart, and the real one | | **judgment, norms, what good looks like** | **people** | ### 00:06:40–00:07:00 # what organizations know | WHAT A COMPANY KNOWS | WHERE IT LIVES | | :------------------------------------ | :---------------------------------- | | how the work gets done | procedures, checklists, habit | | past decisions, and how they turned out | memos, drives, people's memory | | its customers, contracts, and numbers | its systems | | who decides, who to ask | the org chart, and the real one | | judgment, norms, what good looks like | people | ### 00:07:00–00:07:20 # what organizations know - **how the work gets done**: procedures, checklists, habit - **past decisions, and how they turned out**: memos, drives, people's memory - **its customers, contracts, and numbers**: its systems - **who decides, who to ask**: the org chart, and the real one - **judgment, norms, what good looks like**: people ### 00:07:20–00:07:40 # what agents relearn ## preferences - leverage - use - we're excited to - we shipped corrected again today ## where things live - search: pricing sheet - search: pricing v3 - search: final pricing - `finance/2026/pricing.xlsx` found it, again ## how the team works - who signs off on refunds? - which template for renewals? - is Friday's report still due? asked every session ## the biggest model - frontier model - full context, every call - same task as yesterday paying to relearn ### 00:07:40–00:08:00 # rules and patterns **RULES** (live in code) - scripts - skills - procedures - preferences - friction - heuristics - trends **PATTERNS** ### 00:08:00–00:08:20 # rules and patterns RULES: live in code PATTERNS: live in data A spectrum of items: scripts, skills, procedures, preferences, friction, heuristics, trends. ### 00:08:20–00:08:40 # context or weights **GOES IN THE WEIGHTS** - what repeats and holds still: - how this team does the work, - what good looks like here **STAYS IN CONTEXT OR CODE** - what changes: this week's policy - what's permissioned: client data - what you need to inspect: the checks [Line graph showing "total cost" on the Y-axis and "number of runs" on the X-axis. A diagonal line, labeled "explained on every call", represents increasing cost with more runs. A horizontal green line, labeled "learned once", represents a constant cost. The intersection point is labeled "the training run".] ### 00:08:40–00:09:00 # context or weights ## GOES IN THE WEIGHTS - what repeats and holds still: - how this team does the work, - what good looks like here ## STAYS IN CONTEXT OR CODE - what changes: this week's policy - what's permissioned: client data - what you need to inspect: the checks [Line graph titled "ILLUSTRATIVE" showing "total cost" on the Y-axis and "number of runs" on the X-axis. The X-axis has labels "one person's experiment" and "every team, every day". Two lines are plotted: a diagonal line labeled "explained on every call" and a horizontal line labeled "learned once" and "the training run".] ### 00:09:00–00:09:20 # the long way around someone who knows the work -> an expert network -> an environment vendor -> a frontier lab -> the next model -> an agent doing similar work ? [Flow diagram showing a linear progression of six steps, with a dotted feedback loop from the final step back to the first, and a question mark below the loop.] ### 00:09:20–00:09:40 # the long way around - someone who knows the work - an expert network - an environment vendor - a frontier lab - the next model - an agent doing similar work ? [Flowchart diagram showing a linear progression from "someone who knows the work" through "an expert network", "an environment vendor", "a frontier lab", and "the next model" to "an agent doing similar work". A dotted arrow loops back from "an agent doing similar work" to "someone who knows the work", with a question mark below the loop.] ### 00:09:40–00:10:00 # the long way around [Flow diagram illustrating a process with six steps connected by arrows: "someone who knows the work" (highlighted), "an expert network", "an environment vendor", "a frontier lab", "the next model", and "an agent doing similar work" (highlighted). Dotted lines extend from the first and last steps, converging towards a question mark below the main flow.] ### 00:10:00–00:10:20 # the long way around - **someone who knows the work** - an expert network - an environment vendor - a frontier lab - the next model - **an agent doing similar work** [Flow diagram showing a linear progression from "someone who knows the work" through several steps to "an agent doing similar work", with a dotted feedback loop from the agent back to the person, marked with a question mark.] ### 00:10:20–00:10:40 # the long way around [Flowchart showing a sequence of steps: **someone who knows the work** -> an expert network -> an environment vendor -> a frontier lab -> the next model -> **an agent doing similar work**. A dotted line connects the first and last steps, with a question mark below it.] ### 00:10:40–00:11:00 # the long way around [Flowchart showing a process: "someone who knows the work" -> "an expert network" -> "an environment vendor" -> "a frontier lab" -> "the next model" -> "an agent doing similar work". A dotted arrow loops from "an agent doing similar work" back to "someone who knows the work" via a question mark.] ### 00:11:00–00:11:20 # the long way around - **someone who knows the work** - an expert network - an environment vendor - a frontier lab - the next model - **an agent doing similar work** [Flow diagram showing a sequence of steps from "someone who knows the work" through various intermediaries to "an agent doing similar work", with a dotted feedback loop from the agent back to the beginning, marked with a question mark.] ### 00:11:20–00:11:40 # types of learning - **PEOPLE** - decisions - reviews - what good looks like - **HARNESS** - orchestrator - routing - decision model - **KNOWLEDGE** - docs - skills - search - **MODELS** - frontier - open - your checkpoints [Diagram showing four categories (People, Harness, Knowledge, Models) with associated concepts, and two feedback loops labeled "traces" and "checkpoints" connecting "decision model" to "traces" and "traces" to "checkpoints", which then points to "your checkpoints"] ### 00:11:40–00:12:00 # types of learning - **PEOPLE**: decisions, reviews, what good looks like - **HARNESS**: orchestrator, routing, decision model - **KNOWLEDGE**: docs, skills, search - **MODELS**: frontier, open, your checkpoints [Diagram showing four categories (People, Harness, Knowledge, Models) each with three associated concepts in boxes. A line connects 'decision model' (under Harness) to 'traces', and 'traces' to 'your checkpoints' (under Models), labeled 'checkpoints'.] ### 00:12:00–00:12:20 # types of learning - **PEOPLE** - decisions - reviews - what good looks like - **HARNESS** - orchestrator - routing - decision model - **KNOWLEDGE** - docs - skills - search - **MODELS** - frontier - open - your checkpoints [Diagram showing four categories (People, Harness, Knowledge, Models) each with a list of associated terms. Green lines connect 'decision model' to 'traces', and 'traces' to 'checkpoints', which then connects to 'your checkpoints'.] ### 00:12:20–00:12:40 # evals as test cases ## A TRACE - read the request - pulled the account - applied last year's rate - sent the draft ## A JUDGE, OR A SECOND AGENT - the rate was stale - a new case for the set ## THE EVAL SET [Flow diagram illustrating a process from "A TRACE" (a list of steps, one marked as incorrect) to "A JUDGE, OR A SECOND AGENT" (two statements), then to "THE EVAL SET" (a grid of 20 squares, one highlighted).] ### 00:12:40–00:13:00 # evals as test cases ### A TRACE - read the request - pulled the account - **X** applied last year's rate - sent the draft ### A JUDGE, OR A SECOND AGENT - the rate was stale - a new case for the set ### THE EVAL SET [Flow diagram illustrating a process from "A TRACE" (a list of steps, one marked as failed) to "A JUDGE, OR A SECOND AGENT" (a decision point) and finally to "THE EVAL SET" ( ### 00:13:00–00:13:20 # evals as test cases ## A TRACE - read the request - pulled the account - applied last year's rate - sent the draft ## A JUDGE, OR A SECOND AGENT - the rate was stale - a new case for the set ## THE EVAL SET [Flow diagram showing "A TRACE" leading to "A JUDGE, OR A SECOND AGENT", which then leads to "THE EVAL SET". The trace lists four steps, with one marked as incorrect. The judge/agent box explains why the rate was stale and suggests a new case. The eval set is represented by a grid of 20 squares, with one square highlighted.] ### 00:13:20–00:13:40 ## evals as test cases ### A TRACE - read the request - pulled the account - applied last year's rate - sent the draft ### A JUDGE, OR A SECOND AGENT - the rate was stale - a new case for the set ### THE EVAL SET [Flow diagram showing "A TRACE" (a list of steps with checkmarks and an 'x' indicating a failed step) leading to "A JUDGE, OR A SECOND AGENT" (a box with text about a stale rate and a new case), which then leads to "THE EVAL SET" (a grid of squares with one highlighted green square).] ### 00:13:40–00:14:00 # decisions and routing - **Input**: a task from the work - **Processing**: decision model in the harness - **Output options from decision model**: - a frontier model (slow, expensive) - a smaller open model (fast, cheap) - your trained model (fast, cheap, fits the work) - **Resulting metrics**: outcome, cost, latency - **Feedback loop**: outcomes become rewards [Flowchart illustrating a decision model in a harness routing a task to one of three models (frontier, smaller open, or trained model), which then produce outcomes, cost, and latency, with outcomes feeding back as rewards to the decision model.] ### 00:14:00–00:14:20 # decisions and routing - a task from the work - decision model in the harness - a frontier model - slow, expensive - a smaller open model - fast, ### 00:14:20–00:14:40 # decisions and routing - A task from the work - Decision model in the harness - A frontier model (slow, expensive) - A smaller open model (fast, cheap) - Your trained model (fast, cheap, fits the work) - Outcome: cost, latency - Outcomes become rewards [Flowchart showing a "task from the work" entering a "decision model in the harness", which routes to one of three models: "a frontier model", "a smaller open model", or "your trained model". The outputs of these models lead to an "outcome" based on "cost" and "latency". A dashed arrow indicates that "outcomes become rewards" which feed back into the "decision model in the harness".] ### 00:14:40–00:15:00 # decisions and routing - a task from the work - decision model in the harness - a frontier model - slow, expensive - a smaller open model - fast, cheap - your trained model - fast, cheap, fits the work - outcome - cost - latency - outcomes become rewards [Flowchart showing a task entering a decision model in a harness, which then routes to one of three models (frontier, smaller open, or trained model), leading to an outcome with associated cost and latency, and outcomes becoming rewards.] ### 00:15:00–00:15:20 # decisions and routing - A task from the work - Decision model in the harness - A frontier model (slow, expensive) - A smaller open model (fast, cheap) - ### 00:15:20–00:15:40 ## traces to environments ### PRODUCTION - the work, as it happens - real customers, real volume - models in the loop - cost, latency, margins ### ENVIRONMENT - the work, simulated - tasks, tools, checks - every case the model can hit - run again, vary, score [Diagram showing two boxes, "PRODUCTION" and "ENVIRONMENT", connected by arrows labeled "traces" (from PRODUCTION to ENVIRONMENT) and "releases" (from ENVIRONMENT to PRODUCTION).] ### 00:15:40–00:16:20 # traces to environments ## PRODUCTION - the work, as it happens - real customers, real volume - models in the loop - cost, latency, margins ## ENVIRONMENT - the work, simulated - tasks, tools, checks - every case the model can hit - run again, vary, score [Diagram showing two boxes, "PRODUCTION" and "ENVIRONMENT", connected by arrows labeled "traces" (from PRODUCTION to ENVIRONMENT) and "releases" (from ENVIRONMENT to PRODUCTION).] ### 00:16:20–00:16:40 # decision cards ## Claims Triage - Nightly Model **Promote tonight's model?** - Better on four of five suites. - Slower on handwritten forms. **Performance metrics:** - routing - coverage - fraud flags - letters - handwritten ## Pending Decisions - Reply to Northwind about the delayed shipment (drafted) - Which model handles invoice disputes (a/b ready) - Refund above the policy limit (needs a call) - New pattern in failed onboarding calls (12 traces) - Retire the old intake checklist (suggested) [Horizontal bar chart showing performance metrics for different categories] ### 00:16:40–00:17:00 # decision cards ## Nightly Model Promotion - **Promote tonight's model?** - Better on four of five suites. - Slower on handwritten forms. ### Other Decision Cards - Reply to Northwind about the delayed shipment - drafted - Which model handles invoice disputes - a/b ready - Refund above the policy limit - needs a call - New pattern in failed onboarding calls - 12 traces - Retire the old intake checklist - suggested [Bar chart showing performance metrics for a model across categories: routing, coverage, fraud flags, letters (all high performance), and handwritten (low performance).] [Three interactive buttons: "Promote", "Hold", and "Show cases".] ### 00:17:00–00:17:20 # decision cards ## Model Promotion Decision Card ### CLAIMS TRIAGE - NIGHTLY MODEL - **Promote tonight's model?** - Better on four of five ### 00:17:20–00:17:40 # decision cards ### Claims Triage / Nightly Model **Promote tonight's model?** - Better on four of five suites. - Slower on handwritten forms. ### 00:17:40–00:18:00 # decision cards CLAIMS TRIAGE • NIGHTLY MODEL ## Promote tonight's model? - Better on four of five suites. - Slower on handwritten forms. - Buttons: