# Tokens Should Have Jobs — Katelyn Lesse, Angela Jiang — session 2026-07-02T17:45:00.000Z → 2026-07-02T18:05:00.000Z

_60 transcript lines · 0 slides · source: full recording_

## Transcript

This separation is designed thesis of the project, rules for facts, learning for bounded choices, and guardrails for authority. Before selecting an action, the system has established what actually happened. The schema profiler extracts structure, types, nest egg, and null rate statistics. The drift detector compares the current profiler with the baseline and identifies additions, removals, and type changes. The data quality analyzer Checks completeness, validity, and consistency. The error classifier maps log patterns into failure families, and the risk scorerer turns those signals into an operational risk level. These components are deterministic by design. For directly observable data conditions, an explicit rule is easier to validate, explain, and audit than an opaque inference. With richer and representative incident history, some classify Could become learned components. But ML ready is not the same as ML required. The simplest reliable component should own each decision. The policy receives a compact state, failure category, risk level, retry count, drift severity, and data quality condition. It then selects from six actions, retry, covers, rollback, quarantine, escalate, or log. I use tabular q learning because the state and action spaces are small. The queue table is cheap to evaluate, and every decision can be inspected directly. For this state, these were action values, and this action won. Technically, each incident is modeled as single Good morning. We're super excited to be here at AI Engineer with all of you. I'm Caitlin, and I lead platform engineering at Anthropic. And I'm Angela. I lead platform product at Anthropic. And today, we wanna talk to you about a concept that we've been spending a lot of time thinking about and working on with our team, which is this idea that we think that tokens should have jobs. So if you're building an agentic system and you're trying to accomplish some specific outcome, you're trying to get something done with agents, there's one lever that everybody pulls in order to get a better outcome and that's usually Increasing your budget, which means you spend more tokens or you spend more expensive tokens. But we've been wondering, is that all there is? Underlying this assumption of, using the budget is this kind of implicit perspective that every single token is basically fungible. And we've been wondering, is that actually true? Are all these tokens actually fungible? And to test that, we've been thinking, what if we gave tokens jobs? So if you think about the way that you would normally set up an agent to Accomplish a task, you give it that task, you give it this token budget, and then all the tokens that are being spent are basically indiscriminate in the sense that they're all doing one job. They're just executing. But what if you take some of those tokens and they're not just executing, they're doing some other job? So, for example, maybe you take some of your tokens and they're advising the tokens that are executing or maybe the tokens that are executing, try to get something done well and you take some other tokens and you actually grade how well the ex Executor is doing so that it can iterate and try again. Or maybe you have tokens that are dreaming. They're reflecting back on the job that other executors have done and writing learnings to memory so that they can do it again. And what we call each of these, if you take some tokens that are executing and some tokens that are doing some other job, let's call this a strategy. So let's go take a look at the first strategy, the advising strategy. Here, we're splitting up an executor and an advisor. Obviously executes, but crucially, they can call out to an advisor for advice. And then they can take this advice and figure out if they're doing the next step correctly. This is really helpful in use cases, for example, if you're building a sales agent. In an ideal world, you'd have that sales agent be able to actually help the sales rep flag when a follow-up is overdue or a deal is stalling. In this construct, having an advisor to be able to kind of make sure that all the different pieces are actually working is really helpful. So another example is grading. Let's say you're executing And you kinda know exactly what good really does look like. You can define this in a rubric, and then each time an executor tries to accomplish that outcome, you can have a grader provisioned that grades how well the executor did while looking at that rubric. And if the executor did a good job, then great, it can be done. But if it didn't do such a great job, you can iterate again until you get that good outcome. So an example in practice of when you might want to use this is, let's say you have a customer service agent and you're running A store and your customers are writing in and they're saying, I should get a refund for this thing, and your customer service agent needs to be able to respond. You probably have some, like, pretty specific criteria on when you would give somebody a refund. And so what you can do is define a rubric that uses that criteria. You can have a grader that goes and looks at the work that the customer service agent is doing and decide, is it getting it right, and is it coming to the right outcome? And the last strategy we have is dreaming. So in dreaming, there's an executor. Could naturally execute. And then there's a dreamer. The dreamer is actually able to inspect the work and the transcripts of the executor, and then it takes any of the findings that it has, and it writes them to memory. This memory is re picked up by the executor for the next round, so ideally would have improved. A great use case for this is if you're building a recruiting agent. Now recruiting requires a lot of interaction with feedback on whether or not a candidate does or doesn't make sense, and if it's a good fit between both parties. And so by taking all this type of data, if you build a dreaming type of strategy on this It's actually able to kind of sharpen the next round so that it's more and more increasingly useful. So let's make this concrete with some experiments. So what we did was we created a bench of a bunch of tasks related to financial analysis, and And what we're doing with each of these tasks is trying to replicate in the real world, expert human financial analyst, how well would they do on each of these various tasks. And so what we did was we start with a control that's just executing. Let's try each of these tasks, and we'll eval them, and we're literally just But then we can experiment with each of our strategies and see how well we perform. So we start with a super basic experiment. Let's just one shot it. Let's take each of our strategies, and we'll go and just make an attempt to accomplish these tasks, and we'll see how accurate we are. And so you can see here with executing, it didn't do so well, 15% accuracy, but because it was just a one shot, the strategy got to choose how many tokens it would actually spend on its own. And so you can actually see that execute decided And not to spend that many tokens, only 39,000. And as we go into our larger strategies, our more complex strategies, we did choose to spend more tokens, but we did a better job. So this isn't really telling us much because, sure, Dream did really, really well, but it used a whopping 600,000 tokens to get there. That's right. In order to actually figure out if varying the jobs produces any alpha, what we need to do is hold the budget constant. And to do this, we're going to take dreaming's budget, that's 600,000 or so, as the maximum budget that is fixed across the board. And we give every single strategy this budget in order to analyze how well it's performing. And as expected, again, if you give a lot of strategies more budget, you are going to see performance increase across the board. So execute went from point one five to point seven six. Advise and grade went from the sixty's to closer to the ninety's. And that's, again, expected given the fact that if you give things more test time compute, they should generally perform better. But if that was the only thing that mattered, we should actually expect to see execute, advise, grade, dream, actually all Be at the exact same level given the exact same token budget. But what we're actually seeing is that there is an alpha or there is a difference, and therefore an alpha for us to exploit. If you look at execute at those exact same budget level, it gets to point seven six, but Advise is at 0.89. So while minimal, it does exist, and so there is alpha for us to take a look at. Now we decided to take a look at this analysis from a completely different lens. And as Caitlin mentioned, we're doing this bench for a very complex set of financial Task in the real world. And we wanted to analyze the usage of agents with actual experts. So if we look at a financial analyst expert, right, the kind of task that they need to do with an agent is that they're giving it something very concrete, like let's say make a P and L, and then they're getting the result back. Now if that result is 80% accurate on a bench that sounds great, but in reality, what that means for that expert is they have to go back and recompute that P and L themselves, or alternatively send it through Another run. And that's because in this kind of domain, for this kind of task, if you're not 100% accurate, it's actually not useful. You cannot make up an income number, or you can't make up a cost number. Right? You have to make sure that it's 100% accurate. So with this lens of the real world consequence associated with this domain, we needed to recompute our experiments and score them a bit differently. Crucially, we needed to make sure that our experiments had this kind of construct, where if it was scored perfectly, we'd actually give it a pass. And if it had scored anything less than 100% on that kind of Ask, we would actually mark it as a failure. So let's look at a different cut of our data from our experiments with this lens where we're looking for this perfect run 100% accuracy pass. And let's look at what percent of the time each of these strategies was able to achieve a pass. So we've we've got execute, down at 42%, and we've got our more complex strategies doing a bit better up to 75% accuracy. And again, this doesn't necessarily tell us a ton because, you know Each of these strategies might, choose to use different budgets over time. Right? So what we did here was we fixed the budget and we said within a fixed budget, how well do each of these strategies perform? And so what really matters to us actually is if you're trying to get this perfect answer and you're in the real world, you're running a business, what matters to you is the cost to you to get to that perfect answer. And so one way we can think about this is we had our execute strategy, for example. The execute strategy around 40% of this Time will give you that perfect answer. So on average, you can expect to have to run it three times, and you should hopefully, sometime in those three runs, get a perfect answer. And as we talked about earlier, we fixed our budget to that highest token budget strategy, which was 600,000 tokens. So if you spend 600,000 tokens in each individual run, you have to run approximately three times. You can expect, on average, to have to spend 1,800,000 tokens with the execution strategy to get to your perfect answer. And so if we take this analysis and Across the board against all these strategies, this is actually the true cost it took in this domain for that agent to be useful for that strategy. So as Caitlin mentioned, for execute, which is our baseline, this is going to be 1,800,000 true total token cost for you. But Advise, Grade, and Dream are showing us a bit of difference. Crucially, Advise and Grade are actually quite token efficient when you think about the actual usage of the end output of each of these agents. So what does this mean for you as a business? Would actually pick completely different strategies. You'd probably lean towards grade or dream. So if you take away one thing, the thing we want everyone to think about is this idea that tokens are not fungible. You can use your tokens to execute. You can brute force your way through your tasks, and you can throw more budget at it. But if you get really smart about having your tokens do these different jobs and try these different strategies, you're very, very likely to be able to get a better outcome for the task at hand within a fixed budget. And so let's talk a little About how we actually build strategies and how we bring this to life. So we've done a lot of work to create a really excellent harness for individual agents. And if you see this, picture at the bottom here, this is actually the architecture that we've used for cloud managed agents, which is our agentic solution that we give to you within the cloud platform. And what we do on top of this is we start to get into the meta harness level, like the multi agent orchestration and execution level where this strategy can go and be, coordinated between our Executor and our advisor or the other agents within our strategy. And some of these, like dreaming and outcomes, we actually give to you out of the box within cloud managed agents. So with those set of primitives, it's actually relatively trivial for us to construct this kind of, you know, architecture, where we're able to combine these different types of strategies and figure out how they should work together. So for example, it's relatively trivial for us to say, okay, now with this, I can take a task, and I should be able to execute it, but also allow it To advise. And Fable is back online, so we could actually say Fable is the one that's actually advising, the executor. And then I can take all these results and say, send them to a grader so that I can make sure that this is verifying in a loop that makes sense. And if it passes, that's awesome. I want to send all of that stuff to dreaming and make sure that my next run is better than ever. And of course, you don't have to stop there, right? If the right the right primitives are there and the right coordination is there, then you can actually construct really complex setups that fit for all the different types of dynamic That you have. You can invent these kinds of large scale architectures, again, very trivially. And you could also invent completely new jobs, not just the ones of the pieces that Caitlin and I have presented in this conversation. So a big goal that we have over time is to get our models better and better and our platform better and better at dynamically constructing these strategies for you as you're doing work. But in the meantime, as we're working our way there, we would love for you to continue to think about this idea that you should give your tokens jobs and you should Use different novel strategies by combining these primitives in order to get the outcomes that you want for your tasks. Thanks for joining us. Turning my second brain into my living research memory. Let me explain. So within my second brain, I currently have over 5,000 notes in Obsidian and another 5,000 notes in Readwise and some scattered in Notion and Google Drive. And all of this is growing on average with 250 files per month. And this is what I want. On the left, you can see my whole Obsidian vault, this huge mess. And whenever I start working on something such as an article, a new project, a new code base, a new feature, or whatever, I want to actually pull high signal notes that are actually used Full for my current work. And you would ask yourself, why not use directly called Xcode or not BookLM? And the thing is that I am. But you need a system that sits between those harnesses and your second brain. Okay. So let's go back to the root of my problem, which is that I'm always losing my research. For example, my reading list is a graveyard. When I'm scrolling social media and I say that I could expose a a new article, a new new YouTube video, GitHub repository, it doesn't matter. Whenever I actually want to start working on something, I never recall what I have in my second brain, or I have to spend a ton of time actually finding meaningful notes that I can use in my work. Right? And another problem that I have is that I want the system to actually be anchored into my personal notes, into my personal values, into my personal face. I want the system to be personal, to reflect my own thoughts. Right? And that's why in today's video, Luis Francois and I will teach you how to build your own AI research OS. This also comes with code so you can also try it out yourself. And I'm Paul Justine. I'm the founder and CEO of of Decoding AI where I do a ton of content on courses on how to ship air products. And I'm also the coauthor of the LM Engineers Handbook bestseller. And the system, the air research OS that I will teach you in this video is the system that I use in my daily work. And now I will pass the torch to Luis Francois. Thanks, Paul. So I'm Lou Francois Bouchard. I'm the cofounder and CTO of Towards AI, where we build educational courses. And I'm also the creator of, What's AI? A YouTube channel where I explain AI engineering techniques. I used to explain AI research before, now focusing on AI engineering. I'm also the author of the book Building LMS for Production. And, before that, I was a PhD student. So I honestly make research for a living. I use To do a PhD, as I said, in AI and doing tons of research and research work. Now I build courses, I write videos, I research for videos, I build trainings for companies for a living. And all of these things that I do start with a very good research and also leveraging tons of knowledge and insights that we get, at towards the eye from building for clients. So I have tons of notes as well, just like Paul, and we try to Leverage them the best possible. And as you'll see, we've built some sort of tool to leverage our second brain, where as you'll see, there will be some differences between how I use it and how Paul uses it. And that's the core goal of the repository that we built and on this project is that we want you to adapt it for your needs. The whole goal is how can we make research better, but more specifically, how can we better leverage what we have? So let's dive into it. And first, we need to figure out Tool to use and when because this whole research system that we build is not for every query. If you just need a fast answer, like a few quick questions or just something where, that that you would just Google basically. Well, obviously, just Google it or ask ChargebeeD Cloud, whichever system you want. But the problem when doing that is that if you have a lot of following up question or it's a bigger project that you need to build On, and have basically a very long context or tons of information to share, relying on ChargebeeT isn't ideal. And it also means that you are fully dependent on the architecture that OpenAI or ChargebeeT's team built. So the next step here is to ask yourself for a more complex problem, do you need to act quickly or do you want to build some next feature and and do something very difficult? If you just have a small repo for a quick change or Write one article. Just do one thing that you know won't be repeatable that much. Definitely use codex or Cloud Code or some agent that you trust. Sometimes you need to keep on digging to make it better, to improve efficiency, may optimize it more. And so typically, when you have to do that, you want your research sources, your research to stick and to be able to refer to them in the future.

## Slides