# In the Land of AI Agents, the Verifiers Are King — Tariq Shaukat — session 2026-07-01T16:25:00.000Z → 2026-07-01T16:45:00.000Z _59 transcript lines · 39 slides · source: full recording_ ## Transcript I think it's also worth calling out that building is easier, but generating value is still Hard. And I think this is something that we run into, you know, as AI engineers sometimes where we think so much about the process of building and our our setups. But the the point is to generate value. Right? And, there it takes a lot of swings. It takes a lot of tries to find the valuable stuff, but that really is the goal. And that's, like, you know, again, what the world is looking to us to prove that AI can really transform it. So To to end, I just wanted to say, like, go explore, make it real, and, yeah, be less reasonable. Thank you. Please join me in welcoming the chief executive officer at SONAR, Tariq Shalkat. Morning, everyone. You enjoy that last talk? That was amazing. I particularly love the end, the being unreasonable part. I thought that was awesome. I also wanna just I'm trying to calculate the odds of Tarek following Tarek as the first two sessions. In the morning. I think the odds are pretty low on this one. But, thrilled to be here today. As as we just mentioned, I am with Sonar. We are in the code verification space, and I'm here today to talk about verification. And I think we're all here, in large part because we believe to some extent that AGI is here. It's coming. The models we just heard about Fable, it's really incredible what is going on in the in the world today. And yet, we work Almost exclusively with enterprises around the world, and the conversation that we have more is the question mark version. Is AGI here? And why are they asking these questions? It's because you can read the news every day, and I'm not trying to name and shame here, but if you look at KPMG putting out reports that they have to, retract because of hallucinations, doing the same thing, law firms getting into lot. And lots of trouble because of made up citations, made up case law, things like this. I think we can really start to question, how do we get value out of AI? The models are amazing as we just heard, but the hard part is the other Tarek just said is getting value out of it. The struggle is that AI slop is everywhere. I'm sure you all see this inside of your organizations. I'm sure you see this in your everyday life. That AI is amazing. The models are incredible at generating very plausible output. They're incredible at generating things that sound correct, but are they correct? And how do you know that they're correct is a big problem. It's a big problem in professional services as we saw. It's a big problem in legal. But, really, I think if we're honest, it's it's a big problem in every sector, in every field, whether it's marketing or finance or you name it, you have this question of how do you actually know if it's True. How do you know if it's good or if it is slop? And the question that we, we deal in the coding space in particular, we deal with software development. And the question we get as we talk to, I'm sure many of the people here in the room and a lot of our customers is isn't software development different? And we can look at the data on this and, the Mythos models, this is data from, METR. You may have seen this METR. The coding agents are getting better, very quickly. They're getting a lot better very quickly, and you can see, the progression, the exponential curve here. What this shows on this chart is how capable are the models at completing tasks that humans would take. So can they complete a task that takes one hour, two hours, whatever it is. The latest Mythos model, at least per the benchmarking, which was done a month or so ago in the preview mode, was you're getting to sixteen to eighteen hours. So They're actually able the agents are able to complete long running tasks, and it really is starting to transform how work is happening. But the critical caveat when you read the data is this is at a 50% success rate. Okay? So it is again able to complete tasks, but is it able to complete tasks correctly is the question. So if you start looking at let's dial up the accuracy. Right? You dial it up to 80% and there's Still progress, but it is much slower progress. Instead of eighteen hours, you're at about three and a half hours or something along these lines. And by the way, this is still at 80% accuracy. And as I was presenting this to the CTO of one of my, large customers, his response was, but Tarik, I would still put someone who gave me 80% accurate information on a performance review probably, right? This isn't necessarily enterprise grade. The problem is that the Models themselves, and full disclosure, we have not yet, done this benchmarking on the fable models, obviously, because they are just being released. But as you look at the models, the models are getting smarter, but they still produce a lot of problem problematic code. This is benchmarking that we do. We give the models a series of over 4,000 problems, and we basically ask it to generate the response to problems, and then we analyze both the functional correctness. Which is critical, and they all do extremely well on this notion of functional correctness. Right? But then we look at how complex is the code, how buggy is the code, how secure is the code. And what you see with even the state of the art models is that complexity is still high. It's actually quite variable as you can see here. GPT five five is done particularly well on the complexity side of things, it still generates bugs. It doesn't generate massive amounts of bugs, but it still generates And it still generates security issues. So this is the output of the models that are going into the agentic workflows. And, again, this is not you know, I'm at the AI engineer conference. This is not me saying AI is fake or or, incorrect, but it is, trying to address this question of how do you really get value in a production setting out of AI. This is a study that was done in Carnegie Mellon, University, and it looked at what is the actual productivity benefit that you see from the use of AI coding agents. And what you see, I think, really resonates with a lot of what I see firsthand in the market, which is you have a initial just amazing boost of productivity, of velocity in particular. What you see is a three to five x boost in productivity or in in velocity. That Dissipates in three months. At the end of three months, it starts to come back to the the normal before you were using the agents. And if you ask why, it is because of the two pieces in red here that you start to see there's an increase in velocity, but there's an increase in security issues. There's an increase in maintainability issues. There's an increase in reliability issues, and there's an increase in complexity. So, essentially, you're building the technical debt as quickly as you are. Are generating the code or maybe even more quickly, and that creates a different set of work. It creates a different bottleneck. And so to us, this is now the critical question in AI, which is in a world in which code is provable and there are sessions that, I'm actually very much looking forward to attending about formal methods and proofs and things like this. Code is provable. But, when you start dealing with large code bases, software is not, it's still Very complex. It is still very messy. There's lots of, dependencies. There's lots of, technical debt already in most code bases. And so this question of verification is actually key. And what I'm going to be arguing is that you can treat verification as an afterthought or you can bake verification into the process. And if you bake it into the process of generating code, of doing software development, you can actually start to get materially better out. Comes from the coding agents than if you view it as an afterthought, if you view it as just the old school code review. So, as we've been thinking through this, we basically have constructed a framework and there's lots of competing frameworks around this, but I'll just talk you through, ours. We call it the agent centric development cycle. For shorthand, we call it ACDC sometimes, and the idea here is how do you get verification powered agentic loops. At the center, there's a lot of focus on The code generation piece. Like, how do you actually get the models and the agents to generate the code that you need to solve the problem? And what we argue is that you should surround this with the right disciplines, the right tools, the right processes to do three things, to guide the agents, and Tarek was talking a lot about different aspects of this actually, guide the agents, verify the outcomes, and then solve the problems, and you have to make this part of the discipline. Part of the process, part of the new software development life cycle if you wanna be successful in the AI world. So if I double click on some of these pieces, what do we mean by guide? We've done a lot of experimenting around guide. We've just launched a product, yesterday, I think, called Sonar Vortex that starts to get into this area. What we find is critically important is to think about guide as context and constraints. And we separate out context and constraints very deliberately. As context is, you have your code repositories, how do we make it easier for the agents to understand, for the models to understand what is in your code base? If you have a million lines of code, if you have a 100,000,000 lines of code, you have a billion lines of code, the agents work better if they understand your code base. So how do you give it architectural awareness? How do you provide, semantic navigation, maps, and, and help them understand the territory to borrow what Tarek was just talking about? And We find it equally valuable, and I don't think this part is talked enough about to provide the constraints as well. You have guidelines that you want your code to follow. You have dependencies you are okay using, you have dependencies that you are not okay having. You have coding standards, you have guardrails, you have intended architecture. We spend a lot of time talking about existing architecture, but what about where you want to go? And so, this idea of context and constraints. We've found in our testing generates a massive improvement in agent effectiveness and a massive, improvement in token consumption, over 30% reduction in tokens being used to solve a given problem. And and if you ask why it's because you're actually making the life of the agent easier. You're helping it navigate better. So then we get into the heart of this and we really think of guide as preemptive verification. How do you make sure there's less to verify less? To fix this sort of thing, then you get to the heart of verification. And what we believe quite strongly and what we've seen work in practice is this idea of zero trust, multi layered verification. Zero trust, every model has biases, every model produces, has a character, has a personality. So let's make sure we use different models and different techniques to make sure your code is safe and make sure it's reliable to make sure it's secure. And multi layered really speaks to The earlier point that software is complex, software is very messy, software has lots of of of intricacies involved with it. And so what we believe, and again have found to be quite, impactful here is that a combination of algorithmic verification looking at things like data flows, control flows, known patterns, secrets, these areas combined with what is now possible with agentic verification looking at intent business This logic, the unknown unknowns, actually, again, to borrow from the last, presentation. The fusion of these things, the the deliberate multilayered fabric that you put in place can actually you can see the results of this in production. So as we look at our partners and customers who use a multilayered verification approach, they are reporting AI derived production outages being 44% less frequent than the one Who do not. So you can start seeing a material improvement in reliability, in security, and in maintainability. And then the last point I mentioned is technical debt does explode. Right? As you generate code, technical debt is also generated. And, again, this is not stop doing it. This is be aware and let's start controlling it. And so what we, have seen be super effective is to have an Active process to have an active discipline again around code maintenance and thinking about how you do verified code maintenance. I won't walk through every step of this, but it the agents, whether that is a, set of remediation agents, whether it's a strong discipline around verification, does keep your code base clean. And a lot of people have asked me, alright, but do agents care about clean code? Human developers care care about clean code. Do agents care about clean code? And what we find, again, is they absolutely do because the agents have to understand the code base if they're going to operate on it. So this is a one shot view. We think this is something that compounds, but if you just do the exact same agentic tasks on a typical code base and then one that has been cleaned, you see a material reduction in the amount of tokens, reasoning energy, etcetera needed for those cleaner, code bases versus the typical. Code bases. Right? If you make the life of the of the agent easier, if you maintain your code base, then you'll actually see compounding effects. Now, the important thing in our mind is to construct the system. This is how I started is saying, you know, I'm sure all of us do code reviews, you may use static analysis tools, you may use AI code, review tools, a whole range of things. And we believe that you have to put this in a system and again, we're happy to in our booth downstairs. Talk through what this looks like, but we really believe that the construction of the software development life cycle in an AI world, needs to embed this notion of guide, verify, and solve inside of it. And you need to do it in three loops, and you need to think about these three loops. There's the agentic loop, which I think is the key buzzword of the conference. Now but how do you provide the agents as it's generating the code, as it's doing the work with the context and constraints? With the in loop verification so that the agent is getting verification as it's working and how do you fix problems. That's that's the blue loop here. What we what we talk about is the inner loop verification piece. There's the second which is your continuous improvement process. And how do you really combine the power of algorithmic and agentic to generate your your pull request, review the code, and by the way, the velocity of this has to go up massively. So to review the code Using agents and to this multi layered verification, and then you have your evals, and I think the opening speaker talked about how evals may be the buzzword of the, conference. But you have your evals and you have your quality gates to check. Are you actually passing? So you have your your code maintenance loop, agentic loop, CI verification loop, and deliberate design of these loops with verification at the center is a compounding system. It's a system that reinforces itself, and it reinforces itself in the positive, and it reinforces itself in the negative. And we've seen customers who, have kind of neglected as they've rolled out AI coding tools. They've neglected verification. They've neglected this idea of code quality, of code, maintenance, things like that, and you get into a downward spiral pretty quickly. This is what the Carnegie Mellon, case study, or study actually shows. That you actually have all the benefits start to dissipate, or you can get into the self reinforcing loop. And one of the tests we did with one of the large banks who are using some of the cutting edge, the folks who are all around here today, cutting edge agentic coding tools, they can get a 92% reduction in issues if you actually take this guide, verify, solve approach inside of those agentic loops. If, again, this compounds, it's not that each loop is 92 Percent better. It's that as you go through solving the problem over minutes and hours that you actually see a compounding benefits. So that is, essentially how we see the benefit here. They how we see the controlled, value creating use of AI in enterprise settings. And when I say enterprises, people with existing code bases, people with with, you know, millions of lines of code already. There's the agenda. Loop. There's a CI verification loop. There's the code maintenance loop. I'm required by my marketing team to put up a version of this that has our products on here. So these are our products and you can come and see us later, but the most important thing is really to say, our recommendation is this agent the ACDC, agent centric development cycle, the core part is deliberate verification built into the system. So if you'd like to learn more, we have a booth. It's the big red booth downstairs. We'd love to talk. More. We have some double click sessions coming up, so please do, join those, and, have a great conference. Thank you all. Joining us on stage is a member of technical staff at Amazon AGI Lab on Jabbarth. ## Slides ### 00:00:19 > The only way to prove that agents work is to do the best work of our lives — **faster than ever** before. ### Field Guide to Fable Thariq Shihipar / Member of Technical Staff ANTHROP\C ### 00:00:49 # Building is easier, generating value is still hard [Two rows of grey dots, with one red dot in the top row and one red-outlined dot in the bottom row] ### 00:01:20 # AI Engineer World's Fair ## AIE ### 00:01:52 - OpenAI - AI Engineer World's Fair - Akamai - DATADOG [Logo for OpenAI] [Logo for Akamai, featuring a blue wave graphic] [ ### 00:02:25 # AI Engineering World's Fair [Grid of various company logos and names, some partially visible, related to the AI Engineering World's Fair event] ### 00:03:21 # In the Land of AI Agents, the Verifiers Are King ### Screenshot of CNN Business article: "Another 'hallucinated' court filing highlights the difference between Silicon Valley and the rest of the world" Tariq Shaukat / Chief Executive Officer, Sonar [Screenshot of a CNN Business article about a 'hallucinated' court filing, featuring an image of the Sullivan & Cromwell building sign] ### 00:03:53 # Every industry is struggling with AI slop. ## In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer, Sonar (Makers of SonarQube) ### 00:04:19 # But, isn't software development different? ## In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar ©2026 Makers of SonarQube ### 00:04:50 # Coding agents are quickly getting a lot better ## Time horizon of software tasks AI can complete at 50% accuracy In the Land of AI Agents, the Verifiers Are King [Scatter plot showing the task duration (for humans) that AI models can complete at 50% accuracy, plotted against their LLM release date. The Y-axis ranges from 1 hour to 16 hours, and the X-axis from 2019 to 2027. Data points include GPT-2, GPT-3, GPT-3.5, GPT-4, o3, GPT-5, Claude Opus 4.5, GPT-5.2 (high), Gemini 3.1 Pro, Claude Opus 4.6, and Claude Mythos Preview (early), showing an increasing trend in task duration over time.] ### 00:05:24 # Coding agents are quickly getting a lot better at 50% success rate ## Time horizon of software tasks AI can complete at 50% accuracy - 50% ### 00:05:46 # Raise the accuracy bar to **80%** and the progress is slower ## Time horizon of software tasks AI can complete at 50% vs. 80% accuracy In the ### 00:06:19 # The models are getting smarter, but they are still producing problematic code ## Gemini 3.1 Pro High - AVG COMPLEXITY: 158.4 - B ### 00:06:48 # The models are getting smarter, but they are still producing problematic code | Model | AVG COMPLEXITY | BUGS / MLOC | SECURITY ISSUES / MLOC | | :------------------------- | :------------- | :---------- | :--------------------- | | **Gemini 3.1 Pro High** | 158.4 | 614 | 214 | | **Claude Opus 4.7 Thinking** | 171.2 | ### 00:07:22 AI Engineer World's Fair PRESENTED BY Microsoft # The models are getting smarter, but they are still producing problematic code | Model | Gemini 3.1 Pro High ### 00:07:48 # A temporary speed boost + a persistent quality decline - **3 to 5x** - Temporary velocity spike that disappears within 3 months - Increase in lines added at the ### 00:08:19 # A temporary speed boost + a persistent quality decline - **3 to 5x** - Temporary velocity spike that disappears within 3 months - Increase in lines added in the first ### 00:08:51 # Code is provable. ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer, Sonar ### 00:09:21 Code is provable. Software is not. ## Verification is the key to AI success ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube [Sonar logo] ### 00:09:52 # The Agent Centric Development Cycle Verification powered agentic loops ## In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Son ### 00:10:20 # The Agent Centric Development Cycle ## Verification powered agentic loops ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube [Diagram showing a circular development cycle with "Generate" at the center, and "Verify", "Solve", and "Guide" as outer steps, connected by arrows indicating flow.] ### 00:10:47 # In the Land of AI Agents, the Verifiers Are King ## GUIDE Agents need enterprise context and constraints so that they are 'verification aware' ### Repository-aware project specific context... - Architectural awareness - Semantic navigation ### ...and constraints... - Dependency guidance - Coding standards and guardrails - Intended architecture ... provided dynamically and/or embedded in models ### 00:11:18 # GUIDE Agents need enterprise context and constraints so that they are 'verification aware' ## Repository-aware project specific context... - Architectural awareness - Semantic navigation ## ...and constraints... - Dependency guidance - Coding standards and guardrails - Intended architecture ... provided dynamically and/or embedded in models ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer ### 00:11:47 ## In the Land of AI Agents, the Verifiers Are King ### GUIDE Agents need enterprise context and constraints so that they are 'verification aware' **Repository-aware project specific context...** - Architectural awareness - Semantic navigation **...and constraints...** - Dependency guidance - Coding standards and guardrails - Intended architecture **...provided dynamically and/or embedded in models** Tariq Shaukat / Chief Executive Officer ### 00:12:21 # GUIDE ## 30% reduction in tokens used by providing targeted codebase context and constraints Source: Blog post: Cut your coding agent's cost with Sonar Vortex ### 00:12:52 # VERIFY ## Verification should be **zero-trust** and **multilayered** ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief ### 00:13:19 # Verification should be be multilayered, intentional, consistent, and transparent ## Algorithmic verification Repeatable, deterministic assessment based on structural rules and known patterns - Syntax and style - Data flows and Control flows - Structure and Architecture - Known dependency analysis (SCA) - Taint analysis - Secrets ## Agentic verification Deliberate, context-aware reasoning to verify intent and discover anomalies - Intent - Business logic - Performance optimization - Emergent anomaly detection - Deep semantics - State and environment ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube ### 00:13:53 # In the Land of AI Agents, the Verifiers Are King ## VERIFY 44% fewer AI-derived production outages Source: Sonar 2026 State ### 00:14:20 ### SOLVE ## By 2028, AI will create more technical debt than it solves Source: Gartner Market Guide for Technical Debt Management Tools ## In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube ### 00:14:52 # Verified agentic remediation in the CI flow and in the background ## In the Land of AI Agents, the Verifiers Are King - Record Issue - Detect Issue - Generate Fix ### 00:15:25 # Agents care about codebase quality **8%** Lower input and output token consumption on 'cleaner' repositories Source: Does Code Cleanliness Affect Coding Agents? --- ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube [Sonar logo] ### 00:15:47 # Agents care about codebase quality **8%** Lower input and output token consumption on 'cleaner' repositories Source: Does Code Cleanliness Affect Coding Agents? --- ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube ### 00:16:23 # Master the critical verification loops to bring the Agent Centric Development Cycle to life - **Generate** - Generated source code - **In-loop Verification** ### 00:16:51 # Master the critical verification loops to bring the Agent Centric Development Cycle to life ## Agentic Loop - **Context and Constraints** - Codebase understanding and guardrails - ** ### 00:17:20 # Master the critical verification loops to bring the Agent Centric Development Cycle to life The slide presents a flowchart illustrating an Agent Centric Development Cycle with two main loops: ## AGENT ### 00:17:47 # Verification = Compounding loops ## In the Land of AI Agents, the Verifiers Are King ### 00:18:19 # AI Engineer World's Fair ## PRESENTED BY Microsoft # 92% reduction in issues via Sonar Vortex powered agentic loop ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer ### 00:18:48 # Powering the Agent Centric Development Cycle **AGENTIC LOOP** - SonarVortex - SonarSweep **CI VERIFICATION LOOP** - SonarQube - gitar **CODE MAINTENANCE LOOP** - SonarQube Remediation Agent ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer [Logo for SonarVortex, a swirling vortex icon] [Logo for SonarSweep, a brain-like icon] [Logo for SonarQube, a Wi-Fi signal icon] [Logo for gitar, a stylized 'g' icon] ### 00:19:25 # Powering the Agent Centric Development Cycle ## AGENTIC LOOP - SonarVortex - SonarSweep ## CI VERIFICATION LOOP - SonarQube - gitar ## CODE MAINTENANCE LOOP - SonarQube - Remediation Agent ### In the Land of AI Agents, the Verifiers Are King Tariq Shaukat / Chief Executive Officer Sonar Makers of SonarQube [Three-column diagram showing three development loops: Agentic Loop, CI Verification Loop, and Code Maintenance Loop, each listing associated tools with their logos.] ### 00:19:45 # AI Engineer World's Fair AIE