# Process Supervision for Agents That Do Long-Horizon Work — Mitchell Troyanovsky — session 2026-10-14T20:48:00.000Z → 2026-10-14T21:06:00.000Z _61 transcript lines · 31 slides_ ## Transcript Working with you has just been how unimaginably complex and creative the work is, and how it takes a lot of effort, both human and AI, to get things to work. A lot of people in this room who either want to work on the technology side of AI and finance, or perhaps finance side, is there anything that you would mention to them from, here's something that you should know or keep in mind when trying to apply AI to domains like yours? Yeah. I would say it comes down to, we are extremely complex organizations, and it's not Straightforward and, typically, things might look a little bit novel or strange and out of place, but it's probably actually there for a reason or structured in a specific way. I'm sure as you guys have come in, you appreciate that we're super nuanced and complicated. So spending the time and really putting boots on the ground to sit with whatever team or front office professional that you're working to build a tool for or enhance is time well spent. And I think the success and, you know Where we are today with the row space products and kind of being ahead of our roadmap and expectations has been, you know, due to the fact that Michael and the team have spent ample time sitting on the ground with our team, like, pretty much every single day, and, you know, understanding our workflows, which I think is is crucial. -Awesome. Well, I guess last thing I'll say is that, shameless plug, both Rosebase and Apollo Technology are hiring people who want to apply AI to finance, so please reach out to either of us if you are interested. That's it. Jack, thanks so much, and Yeah. Great chatting. Thank you. This is great. Thank you, guys. When AI agents tackle complex tasks, the final answer is only part of the story. To explore how we can evaluate the decisions they make along the way, please welcome Mitchell Troianovsky, cofounder. Of Basis. Hello. Hello. Hello, New York. I'm Mitch. I'm one of the two co founders of Basis. We are based right here in New York City. And the thing that we We focus on is, how can we get agents to do production scale work reliably enough so that we are trusted and used end to end by accounting firms across the country. So a lot of our research really focuses on this. How can you get these agents to do real work in the real economy? So I wanted to start with a question, which is, if you're at this conference honestly, if you're watching This talk, you probably increased your own productivity a lot. You're using co work, or, codex, or cloud code, whatever it is. And yet, if you go look at the average company, the average enterprise, the average firm, their productivity hasn't gone up 10x, 100x. I know my productivity has gone up maybe 100x. But if I were working inside of a company, it wouldn't be anywhere near that order of magnitude. And so why is that? At Basis, one of the things we have is that humans are already used to working with non deterministic systems. It's just those systems are normally their coworkers, not their computers. And so I actually think, to answer this question, it's useful to maybe look at the existing way we organize intelligent systems, which are companies. So how do organizations scale their own productivity? Right? Organizations need to To trust that the people doing the work at every level are doing it in a way that is going to produce good outputs consistently at scale. Right? If you look at this image here, here I'm showing an accounting firm. But this could honestly be any company, traditional company. The person who is the head of the firm, they have literally no idea about the exact day to day of the people at the very bottom. And yet, somehow, they trust the work is going to be done. How does that happen? It does not happen through deterministic systems. Right? People are not deterministic. They are intelligent entities. And instead, there are processes that are put in place. And those processes, if we believe the people who, you know, put them in place think that those processes will generalize to good outcomes, then they can be comfortable that even though the people executing the processes are not deterministic, you will generally get to good outcomes if you follow the process. This is true, by the way, even The smallest step. So in this example, earlier I was showing the process of maybe doing the monthly accounting work for a client at an accounting firm. But even within that process say you wanted to see, how do I reconcile the cash? Well, I need to get the bank data. And even that has a process. The process could be really basic. It could just be, use the approved software to extract the data. But there's still some process that is put in place. You wouldn't expect the firm to just let anyone goog Cool, you know, extract PDF, and then just use whatever they want to go extract it. Because now, you would not have process. Now let us talk about agents. As I said earlier, everyone already has co work, ChatGPT, Codex, Cloud Code but where is that productivity? Our opinion is, there are two main problems that come with what I will call coding style Agent products, where you are working with an agent You just tell it to go and do something. The first problem is that there is a competency gap. Because agents, as you think of them as you think of them today, they outsource the complexity to the user. Right? If I am the user, I now have to manage my skill and memories, and think about how to, when I should use the Agent, I have to have like, theory of mind over it. This is really hard. And for those who are here, maybe you guys. Find it easy. But I promise, for the wide majority of knowledge workers, they have no idea how to do any of this. And there is a second problem, which is, there is a process gap. Right? If you are saying, hey, user, here's your Agent, do what you want, well, now, suddenly, the way the work is being done is essentially up to whoever the user is to customize the way the Agent is behaving, which goes entirely against the way that Organizations scale. The process should be coming from senior leadership down, not from any individual employee deciding how they're going to do their job. Right? If you going back to my example if every employee just made up their own way of extracting data, you know, even back in 2018, that would not be a good outcome. And so we believe strongly that the key to building production agents that actually do work end to end is solving these two problems. You have to solve the competency gap by making It's simple. To the user, you shouldn't even have to know it's an agent. Or you probably know, but you don't have to really feel it. It should be as simple as just working with a coworker. You just tell it what you want it to do, and all the things you expect to work should work. And the second is the process gap. You need agents to be reliable not having a stroke of genius one day and forgetting something the other day. It's not about being perfect. Humans are not perfect. Right? Nobody is looking for Perfect. What you are looking for is reliability and consistency and trustworthiness. So what does that actually look like? We think that to be simple and reliable, where you know you do not have to think about what to do the agent can just do it for you without the user knowing the internals to do it the same way, consistently, you have to be opinionated. And just to show you, actually, one slide back here. I want to point out that getting it right is not actually the main task. Because again, humans are imperfect themselves. The point is to do it in the similar way. It is okay if you can't do something. But you should be consistently able to not do something, and then you get better, just like a person does. If you suddenly, one day, are an expert at something, and then you miss something very basic that would be expected, now you are not reliable. So you have to be opinionated. What's an example of an opinion? Let's say we have an Agent, and we need to go and read a PDF. So, in this case, what is the best way to read a PDF? There's many ways. You can throw it into a VLM, You can go write some Python code and use an open source OCR library. You can work with one of the many OCR specific model providers. You could do an OCR, and then you could have the VLM check its work. You could have five VLMs check its work. You could do a 100 VLMs. And take the best of them. There's literally infinite possibilities here. And so, if you just said, Agent, extract Excel extract PDF, you might get good outcomes, but they're not going to be reliable. Right? Not because the process it did was bad, but because the process it did was not opinionated. So the way we develop opinions is, you use initial intuition. Write some things, you know what the opinions are from expertise. For example, if I'm trying to get an agent to do tax work well, I know that if the agent is going to goes and reads the primary source before returning to the user, it is more likely to be accurate. I don't need a set of evals to tell me that. I don't need to divine that from first principles. We know that. Right? I know that if I have an agent make a PowerPoint, if it takes a picture of the PowerPoint before it returns to the user, it is less like To have a formatting issue that is obvious. And so, there are lots of different ways to develop opinions. And one of the key points here I'll just make, before I move on to talking about how to use the opinions, is that the opinions are not just about accuracy. Right? We just talked about the example of taking a picture of a PowerPoint. Well, that adds latency. Right? You may not want that. Maybe the Agent only makes formatting mistakes one out of a 100 times, and you are okay with that. And so therefore, you do not have to take a picture of the PowerPoint. The point It is not exactly what the opinion is. And this talk, we are going to focus not on how to develop opinions, but why opinions matter, and then how to leverage those opinions as you scale in production. So say you have a bunch of outcome based Evals. And when I say outcome based Evals, I mean what we traditionally think of as Evals. It has a right answer. We have some rubric. We're trying to judge what the agent did. How do you know that the Vows generalize to production. Right? You might have 1,000 cases, or 100 cases. But in production, if you are doing real work, you are going to have hundreds of thousands of situations. And that is only going to increase as the trajectories get much longer, because the amount of decisions that are made in a trajectory just grow exponentially as the long horizon abilities of agents increase. And so, I can't prove, if I if you're looking Distribution of, trajectories. You can't prove every one of them is correct. You don't have a ground truth for every trajectory. But you can potentially look at the trajectories to decide if the behavior, the opinion that you have, is actually happening. And you can then infer that if it's actually happening, it's more likely to generalize to production. It's not perfect. You don't know for sure. But at least you are having an opinion. Let's run through a real example. So let's say that I, for a behavior this is an accounting example where if there is a mistake in the bank reconciliation, I don't want the agent to, like, post an entry to make a tie. I want it to sort of explain the difference, instead of putting a plug. Again, this is subjective. You might want the other way. But what you don't want is a customer getting one way One time, and another way the other time. And so, what would you do? Well, first, let's understand what governs the behavior. Right? The behavior is not just, you write the two sentences into your system prompt and it just works. That might work for very simple, narrow agents. But when you're building these big agent systems, the behavior is actually a combination of the system prompt, your progressive disclosures, potentially tool descriptions, obviously, the training of the model. There's a lot of things that govern the behavior. And one tip, by the way I have for when you think about building production agents is, really think of the context the agent is going to see as it's working as runtime training data. The word prompt, I think, trivializes it a bit. Right? At the end of the day, this is the data that the Agent is being trained on at runtime not literally trained on, but effectively trained on. And so you want to care about every single token and understand how that data causes the Agent to behave in certain ways. So, when you're Putting behaviors into production. It might be very difficult for one behavior to put it in exactly one, you know, paragraph that you know exactly, because, again, the behavior is all of the data at once, not just one thing. And one of the things that we developed that basis was this concept of a behavior spec. And the idea of a behavior spec is essentially saying, before we update the context, or write the context, what do we actually want? What is our intent? You can kind of think of this a little bit as, like, a product spec for the agent era. And you can't write you shouldn't be writing every detail down. You should be taking the most important opinions and putting it. Because if you put everything, you'll be duplicating a lot of states. So you define the ones that matter the most. Most. And then you can use them as a way to align your team, from a spec perspective. But then you can also use them as a rubric. And you can say, in this particular trajectory, whether that Is in any trajectory that does not have a ground truth, did the agent get into a situation in which this behavior was relevant? And then if so, did it actually exhibit the behavior? And this way, you can scale the amount of signal that you get on your trajectories far larger than trying to increase the amount of outcome based Evals, Ground Truths, and Rubrics that you have. Those are really important. But you cannot scale those anywhere near as fast as you can scale the ability to supervise what the Behaviors of the agent are. Let us just take a quick example here. This is actually the second concept, which is called forking, which we can combine here with behavior specs to think about. Forking if you have ever used Cloud Code or Codecs, the Web Apps, there is this little fork icon under each message, where you can essentially create a copy of the Agent using the same thing, which works really well on your local computer. It is hard to build a system that does this in the Out perfectly with recreating the database environment and recreating the, the prompts, all the different environments. But forking is another tool that we use to check the behavior. So a flow that you can imagine is saying, we had a trajectory could be in an eval set, it could be in testing or in production and it failed the behavior eval. And now you're trying to understand why. Why did it fail the behavior eval? Why did it not exhibit The behavior. And then you can go into the trajectory, go right before the point in which the agent made the decision that was in contrast to the behavior assuming, again, your behavior is good, and it is actually producing signal and then you can fork it. And when you fork it, you can ask the agent, or you would fork it right after it made the decision. You could ask it, why did you do that? Or you can go and update the context of the agent, and see if that improves the behavior. So with forking, you can start to actually take the signal and start to figure out, okay, how can we then improve the harness to now leverage the signal? Versus, if you're just looking at signal from an outcome based eval, you can't do that anywhere near as easily, because it failed the outcome. How do you know why? You have potentially 1,000 Inference calls in your five hour long Agent trajectory run. And then we have the last tool that I would like to talk about this is actually my favorite which is called Simulations. This is going to sound silly, but I promise it's very effective, which is that you can, when working with coding agents, ask the coding agent to make a sub agent that pretends to be your production agent. And you have to tell it to not cheat. And you give it the system prompt and the skills, and you render everything perfectly. And what you can do is, you can ask it what it would do in some scenario. You can give it the situation. And see if you can recreate the situation that you saw in your Eval or in production. And if you can recreate it, now you can tweak your context to ## Slides ### 00:01:34 # PRESENTED BY Arize [Abstract graphic design with geometric lines, resembling an art deco cityscape] ### 00:01:57 # AI Engineer ## NEW YORK PRESENTED BY arize [Abstract line art illustration of a stylized cityscape or skyscrapers] ### 00:02:34 [Abstract geometric patterns with glowing lines, resembling stylized cityscapes or light trails.] ### 00:03:00 # Process supervision for **long-horizon** agents How to build reliable agent systems Mitchell Troyanovsky Co-founder, Basis [Logo for Basis] ### 00:03:35 # Why hasn't agent productivity *scaled*? ## Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis ### 00:04:33 # How do organizations scale? ## An accounting firm > How can the head of the firm trust it'll work? ## Its clients - Dental practice - SaaS startup - Coffee ### 00:05:00 # A process the firm signed off on - Head of the firm: Signed off - Process: Categorize -> Reconcile cash -> Explain changes -> Review ### Clients - **Dental practice ### 00:05:31 # Even the smallest step *has a process* - **Month-end close process:** - Categorize - Reconcile cash - Explain changes - Review - **Cash reconciliation sub-process (part of Reconcile cash):** - Get the bank data - Match to the books - Chase differences - Workpaper ### Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Flowchart illustrating a financial reconciliation process, with a bank statement document feeding into the "Get the bank data" step] ### 00:06:05 # Okay, now enter agents. ### 00:06:34 ## Coding-agent-style products outsource complexity to the user user of mind over an alien intelligence. Competency gap ### Process Supervision for Agents That Do Long-Horizon Work Mitchell Troy ### 00:07:05 # Coding-agent-style products outsource complexity to the user of mind over an alien intelligence. ## Competency gap - Slop cannon - Not productive - Really good ## Process ### 00:08:05 ## What does it take to build **production agents**? Solve these two gaps - Competency gap -> Simple - No agent whisperer required. - Process gap Basis ### ### 00:08:32 # What does simple and reliable look like? ## Simple - No opinion needed. - No special prompt. - It just works. ## Reliable - Not just this ### 00:09:04 # Getting it right *isn't* the task - Can this client deduct their home office? - Agent - From pretraining - From Wikipedia - From the primary sources - Right answer - At Basis: primary sources before any tax conclusion. ## Getting it right reliably, at scale, *is*. - Home office - Rental property - Stock sales - Partnership K-1 - Moved states - Crypto sales - Situations you haven't seen yet [Flowchart showing an agent processing a query by drawing from pretraining, Wikipedia, or primary sources to find the right answer] [A series of dot-based icons representing different tax-related categories, with varying numbers of dots, and an empty set of circles for "Situations you haven't seen yet"] ### 00:09:33 # An example opinion: *reading any PDF* - **Simple** The user never chooses how. - **Reliable** The same rule on every PDF. ## Process Supervision ### 00:10:03 # An example opinion: *reading any PDF* Simple The user never chooses how. Reliable The same rule on every PDF. ### Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Flowchart showing a process for PDF analysis: A PDF document `mar_statement.pdf` is fed into a step to "Infer its type from the name". This branches into two paths: "Semi-structured" leading to "OCR A", and "Structured" leading to "OCR B". Both OCR A and OCR B outputs then go to a decision point asking "Tables or diagrams OCR may miss?". If "Yes", the process proceeds to "Also verify with a vision model", which is noted as "More expensive".] ### 00:10:31 # How do you develop **opinions**? - Outcome evals - What actually works - Your targets - **Latency** - **Cost** - **What the model can do** - **What you're OK with** - Your situations - Only bank statements (Specific approach) - Any PDF (General approach) ## Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder [Diagram showing six document icons with checkmarks and one cross mark below them, illustrating outcome evaluations.] [Diagram showing a horizontal line with two stacks of document icons at either end, representing "Only bank statements" (specific approach) and "Any PDF" (general approach), with "Us" in the middle.] ### 00:11:04 # How do you develop opinions? Outcome evals - What actually works Your targets - Latency - Cost - What the model can do - What you're OK with Your situations - Only bank statements - Specific approach - Us - Any PDF - General approach ## Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Diagram showing five document icons, four with checkmarks and one with an 'x', representing evaluation outcomes.] [Diagram illustrating a spectrum of situations, from a specific approach (only bank statements) to a general approach (any PDF), with document icons at each end.] ### 00:11:31 # How do you develop *opinions*? ## Outcome evals - What actually works ## Your targets - Latency - Cost - What the model can do - What you're OK with ## Your situations - Only bank statements (Specific approach) - Us (Any PDF, General approach) ### Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Six document icons, five marked with a checkmark and one with an 'x'] [A horizontal diagram showing a progression from a single document icon labeled "Only bank statements" to three document icons labeled "Us" and "Any PDF"] ### 00:12:04 # How do you know your outcome evals generalize to production? The evals are the only ones with true ground truth ## Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Bell curve diagram showing "Situations in production" on the x-axis, with a central region labeled "Your outcome evals" and vertical bars indicating specific "A situation in production" points.] ### 00:12:37 # How do you know your outcome evals *generalize* to production? The evals are the only ones with true ground truth I can't prove this one is right. I can ### 00:13:05 # Here's a behavior we want > When a reconciliation doesn't tie, the agent investigates and explains the difference. It never posts an entry just to make it tie. Cash rec ### 00:13:33 # What actually decides the behavior? - **System prompt** - You prepare work that ends up in clients' books. - Deliver work a reviewer can sign off on: complete, ### 00:14:05 # What actually decides the behavior? - **System prompt** You prepare work that ends up in clients' books. Deliver work a reviewer can sign off on: complete, reconciled and explained ### 00:14:38 # What actually decides the behavior? - **System prompt** - You prepare work that ends up in clients' books. - Deliver work a reviewer can sign off on: complete ### 00:14:59 # Behavior specs 1. Align on the opinions. A forcing function, for us as builders. 2. Define the ones that matter most. Apart from the prompts, skills and tools. Open-sourced with Braintrust · [agentbehavior.dev](agentbehavior.dev) ## Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Diagram showing multiple builders contributing to a single "Behavior spec", representing "One opinion, agreed"] [Diagram comparing "What the agent reads" (System prompt, Skills, Tools) with "What we want" (Behavior spec)] [Three buttons labeled "Behavior specs", "Forking", and "Simulations"] ### 00:15:33 # The spec and the judge Example - Behavior specs - Forking - Simulations ### Behavior spec When a reconciliation doesn't tie, the agent investigates the difference and explains ### 00:16:01 # Forking ## Example ### The flagged run Followed? **No** Saved point just before it decides what to do about the $412.06 ### 00:16:33 # Forking (Example) ## Change the harness - **Month-close skill** - "...record adjustments only for differences you can explain with a source." - **System ### 00:17:05 # Forking (Example) - Behavior specs - **Forking** - Simulations ## The flagged run - Followed? **No** ## Change the harness - Saved point ### 00:17:34 # Forking ## Example - Behavior specs - **Forking** - Simulations ### The flagged run Followed? **No** Saved point just before it decides what to do about the $412.06 ### Change the harness - Month-close skill: "...record adjustments only for differences you can explain with a source." - System prompt: Add "Never force a reconciliation to tie." - Review tool `check_reconciliation`: "...PASS when every difference is explained by a sourced item or flagged for the reviewer." ### Forks and Followed? - **Forks x 5:** Sometimes. The review tool still fails anything that doesn't tie. - **Forks x 5:** Sometimes. For the same reason. - **Forks x 5:** ✓ Yes. No side effects. Iterate on the harness. **The fix was the tool, not the prompt.** --- ### Process Supervision for