# Your Judge Is Hallucinating Too — Brian Kennedy — session 2026-11-11T23:06:00.000Z → 2026-11-11T23:19:00.000Z _33 transcript lines · 25 slides_ ## Transcript Just the categories alone and uses gel. So whenever a new email comes, it only needs gel. It just looks at the, looks at the email and fits the categories, and we basically have a list of follow through. So if this matches this category, you are gonna get an email. This team is gonna get an email, or that team is gonna get an email so easily. And we basically removed LMM completely from the picture on this one. So... And we saw a humongous speed increase on on just email processing. It was almost At real time, an email comes and suddenly somebody will get notified. So that was an amazing, amazing thing to see as well. So over the few months, we were so excited to play around with Jev. And every day, the Slack threads would light up everywhere in the the company that I found this pattern. I found that pattern. And, it was such an exciting time, during these last three months. And, I wanted to share that, there are so much more patterns and strategies to be found. We are only Scratching the surface of this. And, I want us to, try out and, like, share the strategies that we all find and collaborate here. And if you're interested in trying out UiPath flow, we have it available for free right now. Scan the QR code on the screen here. And we'll be able to... It'll take you into a portal where you can basically point your coding agent at it and get complete access to build your first automations there. Thank you for your time, everyone. Thank you. Alright. Good afternoon, everybody. I am Brian Kennedy. I'm an engineer on ZoomInfo's Agentic platform. Now I want to start with showing you something one of our agents wrote. This is a real output from our product. The agent's job was to schedule an email for a customer. And Is what it reported back. It said it scheduled the email. It queued it up and that it was gonna go out on Tuesday. We don't just trust the agent though. A second model checks the agent's work. I'm gonna call this one the judge. The judge read this, and it accepted it. Great. Now, here's the problem, and this is what had actually happened. There, these are the tool results, the record of what actually ran when the agent called the tools. I'm going to call these the receipts. The agent tried to create a record in Salesforce, and it failed. The error message that came back says the record was not created, and it tells the agent not to report a success or even make up a record. We wrote this message ourselves for exactly this situation. The agent tried again. It created a task, which is a to do. No email was ever scheduled. The agent doesn't even have a tool that can send an email. The agent made it up, and the judge approved it. So that's two failures. The agent fabricated a send, and the judge rubber stamped it, approved, a claim that never actually checked. So if you build agents, you probably have run into this, and you probably have a judge like this. It might be a code reviewer, or a grader of your evals, or a supervisor in your loop. Anytime a model decides whether another model's work is good, this can happen. But for the next ten minutes, I'll show you how we caught this and what we did about it. First, some quick background. My team at ZoomInfo, is called Agent Teams. We announced this last week. It's a multi agent orchestration layer that lives in ZoomInfo's GTM studio. It coordinates a team of specialized agents across a company's systems to run GTM workflows. For example, find companies, research them, write an email, log it into their CRM. Each agent has its own roles and instructions, like... And tools, like Salesforce, HubSpot, Slack. These agents make Real, decisions and take real actions in a customer's systems. They write records, and they send messages. Every team and every agent finishes a step. A second model checks its work. That's the judge, and it has three options to move on, to redo the step, or to stop the job altogether. You've probably heard of this called LLM as a Judge. It's the same idea as a code reviewer or an eval grader. Now, the hard part For us, it's one model, and it has to grade every kind of agent that we're running through it. Our customers build their own workflows with any agent, in any tool, in any order, so the permutations are almost infinite. So the list of jobs for the judge has to grow with it. The judge can tell whether the work looks finished, but what it can't tell is whether the work was right, because that depends on what right means for each job, and we're asking it the same question for all of them. So here's where we're going. First, I'll talk about what we found when we checked the judge, then what we chose instead, and what you can do today with your own agents. To start, we ran offline evals. We took 500 real steps from production, each with a decision that the judge had made. Then we took a larger frontier model and regraded every one of those to come back with an accept. Or send back. Then we hand checked each of those samples, and we checked the grades. Where we disagreed with the model, we win with our call. This gave us a reference label for all 500 steps. And it turned out the judge was right 65% of the time, so more or less a coin flip. It accepted a 118 steps that it should have sent back. That's more than one in four of everything that it accepted. Pretty bad. And when we read through those, the most common problem was that, similar to the email at the start, the agent reported that it did something, like send a message, and no tool call had ever done that. So to see why the judge had accepted it, let's go back to the email example and look at what the judge was given. On the left, you have with the agent's own summary. Think of it as like a story. It's confident, it's well written, it says the email was scheduled. Now, on the right, you have the receipts. This is the actual tool results. There are air... Errors, failed to create messages, telling the agent not to make up the record. Now, the judge had both of these, and it went with the story. The work looked fine on the surface, so it accepted it. Now, this isn't only our judge. You can think of a code review agent that reads the PR description before it looks at the diff. It's a similar kind of problem. If the description says, I The bug, added tests. The agent is gonna lean on that when it reads the code. So the obvious fix at the start was to tune the judge and write a better prompt. For us, that was gonna be a significant amount of work. Our agents write sales emails, update CRMs, do research. And to judge each of these well, we'd have to figure out what it was that the user was asking, give the judge the right rules for every kind of work, and keep it all up to date. That's basically a custom judge for every use case. What we went with, we went in a different direction. We stopped asking one big general question, is this good? And we started asking a short list of small yes or no questions. They are the same questions for every step, and each one is small enough to answer by looking at the receipts. The first option here is basically a prompt tuning project that never ends, whereas the second one is a small handful of small checks. Now to answer those questions We use JEV. You've heard a lot about JEV today, so I'll skip that and keep that short. It's a tight decision model, not a generative model. For mom at home watching, just ask Claude. And we'll send it structured facts, you know, about the step and a list of typed questions. A typed question is a yes or no question or a choice with... From a fixed list of options. It sends back a probability for each answer, and it answers all the questions in one call. We gave it four things, the instructions, the, input, the receipts, meaning every tool call, and whether it was successful or failed, and the agent's report. Then we asked for one choice, which was accept, redo, or stop, and that choice makes the decision. But then we also asked seven yes or no questions. Those seven questions tell us the why. They decide between a redo and a stop, and ## Slides ### 00:00:24 # Mining: emails become structured data LLM designs. Jev judges. Code checks and counts. Code flags issues. Definitions improve. Synthetic mailbox. Category spreads are illustrative; the assign totals ### 00:00:52 # Mining: emails become structured data LLM designs. **Jev** judges. **Code** checks and counts. In UiPath, a Flow classifies each new email with Jev and routes it. Synthetic mailbox. Category spreads are illustrative; the assign totals were measured once on the sample mailbox, Jev stage only. Not a benchmark. Beat 5 is a screenshot of the Email Routing flow in UiPath Studio Web. ## Your Automation Is Overthinking It Kevin Bernstein / Solution Architect UiPath [https://uipath.com/](https://uipath.com/) [Screenshot of a UiPath Studio workflow diagram showing email classification and routing steps] ### 00:01:24 ## WHAT WE LEARNED # Many more patterns to be found out there. Don't overthink it. 1. **Find a repeated question**: Defined input, and an answer the next ### 00:01:52 ### WHAT WE LEARNED ## Many more patterns to be found out there. Don't overthink it. 1. **Find a repeated question** Defined input, and an answer ### 00:02:21 # Code Summit ## AI Engineer ### 00:02:57 # LIVE TRANSLATION //AVAILABLE Scan the QR code to access real-time translation. AI Engineer Code Summit Presented by Google DeepMind [QR code for accessing real-time ### 00:03:22 ## AI Engineer Code Summit # STAY IN THE LOOP SCAN THE QR CODE OR VIEW THE SCHEDULE AT AI.ENGINEER/SCHEDULE [QR code linking to https://ai.engineer/schedule] ### 00:03:57 [Dark background with a subtle, wavy pattern of small grey dots] ### 00:04:18 # AI Engineer Europe 2027 ## ExCeL London ## 02.17 - 02.19 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. [ai.engineer/europe](ai.engineer/europe) [Black and white photo of an ornate building in London] [Black and white photo of Big Ben and Westminster Bridge] [Black and white photo of the London Eye] ### 00:04:53 ### AI Engineer Code Summit # STAY IN THE LOOP SCAN THE QR CODE OR VIEW THE SCHEDULE AT [AI.ENGINEER/SCHEDULE](https://ai.engineer/schedule) [https://ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to ai.engineer/schedule] ### 00:05:53 # AI Engineer Europe 2027 ## ExCeL London ### 02.17 - 02.19 2027 Europe's flagship technical ### 00:06:21 # Your Judge Is Hallucinating Too AI ENGINEER CODE SUMMIT · SAN FRANCISCO · 2026 Brian Kennedy @dans_la_cuisine [Zoominfo logo ### 00:06:52 # WHAT AN AI AGENT REPORTED · A REAL STEP OUTPUT ## THE AGENT'S REPORT Scheduled and queued the **personalized outreach email** in Salesforce, linked directly to the contact ### 00:07:25 # SAME STEP • WHAT THE TOOLS ACTUALLY RETURNED ``` salesforce • create status: error "The record was NOT created; do not report this step as successful or fabricate a created ### 00:07:52 # AI Engineer Code Summit ## Your Judge Is Hallucinating Too The agent fabricated a send. The judge rubber-stamped it. Presented by Google DeepMind Brian ### 00:08:25 ## A salesperson builds a small team of AI agents to do a multi-step job. - **find** -> **research** -> **write** -> **log** each step = one agent ### 00:08:52 # After every step, a second model decides if the work counts. "LLM-as-judge." If you run agents, you probably have one of these. ## Your Judge Is Hall ### 00:09:25 # One judge. Every kind of work. any agents • any tools • any order ## Your Judge Is Hallucinating Too Brian Kennedy / Principal Engineer, zoominfo Presented ### 00:09:54 1. What we found 2. What we chose instead ### 00:10:25 # We ran an offline eval 500 production steps → larger model re-grades → we check by hand Result: a reference label for every step [Flow diagram illustrating three sequential steps: 500 production steps, followed by ### 00:10:56 # 500 PRODUCTION STEPS • THE JUDGE VS THE REFERENCE LABELS - **The Judge Was Right:** 65% of the time - **Bad Work It Accepted:** 118 (more than 1 in 4 of its accepts) The most common miss: the report claims an action that no tool call performed. ## Your Judge Is Hallucinating Too Brian Kennedy / Principal Engineer, ZoomInfo [QR code] ### 00:11:23 # WHY? • WHAT THE JUDGE WAS GIVEN ON THE OPENING STEP ## THE STORY "Scheduled and queued the personalized outreach email ... for automatic outreach execution on Tuesday." ## THE RECEIPTS - `hubspot · error` - `salesforce create · error` > "do not ... fabricate a created record" - `salesforce create Task · success` ### Your Judge Is Hallucinating Too Brian Kennedy / Principal Engineer zoominfo [QR code linking to https://www.zoominfo.com/] ### 00:11:55 # Your Judge Is Hallucinating Too ## THE OBVIOUS FIX ### Tune the judge - A judge per kind of work - Criteria for each use case - Prompt tuning with no end [https://zoominfo.com](https://zoominfo.com) ### 00:12:19 ## Your Judge Is Hallucinating Too ### THE OBVIOUS FIX **Tune the judge** - A judge per kind of work - Criteria for each use case - Prompt tuning with no end ### WHAT WE DID INSTEAD **Ask smaller questions** - Same questions, every step - Answered by the receipts - Seven yes/no questions Brian Kennedy / Principal Engineer zoominfo [https://zoominfo.com/](https://zoominfo.com/) ### 00:12:52 # We used a typed decision model, not a generative model. ## IN Structured facts about the step, plus a list of typed questions ## OUT A probability for each possible answer > Jev, from TypeSafe ### Your Judge Is Hallucinating Too Brian Kennedy / Principal Engineer Z zoominfo [https://zoominfo.com/](https://zoominfo.com/)