# Does Code Quality Even Matter? — Max Kanat-Alexander — session 2026-10-13T13:54:00.000Z → 2026-10-13T14:12:00.000Z _69 transcript lines · 35 slides_ ## Transcript And we consider evals to be kind of the same thing. If models are changing, that's like a runtime dependency that your application has. And if your agent is gonna change models, it doesn't know without the evals what's going to happen to its to its outputs and to the quality of those outputs. So again, one tempting answer here is let's have some kind of central evaluation methodology that waits for a full eval suite and a committee sign off. But now every like prompt tweak, every one line change is gonna sit in that same queue. So we don't wanna put more decisions on more reviewers for things like that. We we don't wanna trade away that iteration. So again, you wouldn't ship software without unit tests, so agents need evaluations for the same reason. And we want to provide this stuff at the Of the firm that's going to be the same for everybody. So team a writing evals, team b writing evals. They're both going to have many of the same features. So we want to have one shared approach to how we do evals and have the shared approach be something that's run by the firm and available to everybody for doing their evals. So that covers all the common parts of evals, and all you've gotta do is just write what true about your agent. And finally, maybe you've had the experience of being asked to approve lots and lots of individual agent action. Is when there's a human in the loop. So that's fine at a small scale, but once you're operating agents in parallel or in production, this becomes completely onerous and review requests come in faster than anybody can be expected to approve them. I mean, after the first few times that it's happening, you succumb to review fatigue where you just see yes, yes, yes, yes, yes. And eventually, you click yes on something that's, you know, activating the new... The nuclear bomb. Right? So you don't want to click yes by reflex, but that's going to happen inevitably. And one recent example of this This is, earlier this year, a director at an AI research lab told her Open Claw agent to suggest emails to delete from her inbox and then wait for her approval. But her inbox was so large that eventually, the agent contacted its context and lost the part that said, wait for my approval. And then it deleted hundreds of emails and didn't stop when she told it to stop on her phone. So again, you might be tempted here to make a human approve every action, but that's not the right way to do it. A better approach is to let these independent conservative classifiers make decisions. About routine calls, and then send a person only the requests that are most important, the ones that are most likely to make a difference. If you're saying yes 99.99999% of the time, then that's probably just a rule that the agent should follow, not a request you need to ask the human for permission for. And the stuff that's unclear or ambiguous or potentially dangerous, yes, surface that to a human. Or if it's definitely just dangerous, deny it outright. So this is a way to not have to do all the review work all the time. So we don't wanna take the same approach for both explicit, when explicit rules apply. We want deterministic fast paths. When explicit rules don't apply and there's some ambiguity, we want a classifier to do that work instead. So putting those five pieces back together, you create a registry so that every agent's visible from the day it's created. You score each agent so human review goes in the right spot. You run agents on guardrails so that they only touch what they need. You share evals and metrics so everybody's reaching the same bar without having To reinvent the wheel. And then finally, you govern your run time to make sure that classification happens there and not somewhere else. So each of these controls works together. So I'll leave you with one question. Why do cars have brakes? And it's not just so that you can stop. It's so that you can go fast and stop safely. So So controls for our agents works the same way. Thank you very much. I'm John Camanello, and I hope you had a good time. Here to help us understand the missing principles around code quality and finding a path through competing viewpoints, Please welcome Max Kanat Alexander, principal software engineer at Capital One and author of Code Simplicity. All right. So today I figured I'd just pick a nice, uncontroversial topic that nobody has any opinions about. Right? Nobody nobody's said anything on the about on this about social media, you know, and of course, every take about this on social media is, like, really deeply new. Wants, you know? Like, people have lots of, like, depth and, like, you know, really big thoughts. You know, maybe there's a little bit more to it than, like, yes or no. And, you know, of course, if you ever ask a senior engineer a question, we always have the same answer to every question, every senior engineer's favorite answer to every question which is, it depends. Now, that's not a very helpful answer, though, frankly, because what does it depend on? So let's start with this. Why did code quality ever matter in the first place? Like, I see a lot of people be like, well, it made me feel good. That's not really a sufficient justification. So let's see if we can inject some nuance into this conversation and help everybody at least be able to have a basis for reasoning about it while we're discussing it. So let's see if we can make a framework here. And that's what we're gonna try to do with this talk. Okay. So we're gonna start off with everything you want to ever accomplish with a piece of software, depends on two factors and two factors alone. And that is understanding and execute. On that understanding, whether that's aligning a model, scaling a system, driving adoption, anything, all software success is determined by these two factors. This is why we have things like debuggers and observability and evals and even tests. All those things actually are there to improve your understanding or ability to execute on that understanding. People think that they're there for other reasons. But fundamentally, they're there for that reason. Everything we have ever done in the field of code quality is exclusively and only for these two things. It's not for Else. And that doesn't have something to do with correctness or something else. It has to do only with these two things. Now there's two understandings that are the most important to have if you're going to be a software engineer. There are other understandings to have about software if you're gonna be a product, if you're gonna be in sales. But if you're a software engineer, there's the two that are the most important. And the first is, what is the behavior of the system? Like, con the concretely, not what does a doc say the behavior of the system is, but what is the actual behavior of the system? And the second one is, how do I correctly and safely modify that Behavior. Okay? On a very small system, if you want to understand what the system does, I just tell you what it does. It prints hello, world. You say, I want it to say hi, world. I say, okay. I will make it say hi, world now. There's very little to understand about it other than purely what it does. That's all of the understanding that you need. However, any real software system is enormously complex. This is a real map of the codebase of Chrome. Which is the open source project behind Chrome. And if you think about all of the different ways that these components can interact, they have essentially infinite possible behaviors. Like, I could not describe to you in words the behavior of Chromium as a whole, as a system. However, what I can describe to you... And as we all know, everybody's been a software engineer for a while. I can describe to you components of the system. I can successfully Describe to you how a component of the system behaves. So this is the URL parser the actual functions and stuff for the URL parser of Chromium. It figures out all the different parts of the URL. And this is the whole reason that code exists. The whole reason that we have code at all in the first place is for structure. It... Because it... What do I mean by that? Okay. So both of these two things on the left and on the right Will print hello world if we're on a Linux machine. However, on the left, you can see that it is hello world. It has the identity and structure of hello world. On the right, you just see a series of instructions with no particular identity recognizable or understandable identity. It's not a component. It's just a bunch of instructions. While that's understandable there, if that was millions or billions of lines of that, it would be much harder to grok. In the past, we've talked about Things like readability or testability or things that seemed like human concepts about code. But what actually mattered the most always was that the system had some sort of understandable structure. So it's the structure that actually matters. And when this modifies when this matters the most is when you want to modify the system, the second, understanding that we talked about. All modifications to a system carry two categories of risk. The first is that you will introduce some sort of problem. Problem isn't just a bug. It's anything bad that could happen. Like your project could take much longer than you expect. It could not behave in the way that you wanted it to for your users. The project could fail because you have budget overruns. Any kind of problem could happen from just you trying to go through the process of making this change. And the second is you could make it harder to make future changes. In particular, what that means is you could make it more likely to have future problems. You could increase future risk of the system. These two things Are caused by under... Problems with understanding. So the first has to do with the difficulty of understanding the system in the first place. And the second is you make the system harder to understand. I can reduce every post mortem I have ever read for any incident to these two factors in one way or another. The production configuration was too hard to understand. Somebody made something impossible to understand. There were emergent properties of the system that were impossible to understand. Whatever. Point two Feeds into point one in a vicious cycle that compounds in a nonlinear fashion. So the difficulty of understanding does not grow linearly. It compounds over time. You will eventually reach a point and all systems eventually tend to reach this point where every modification involves some sort of problem. And, you know, some some systems get to the point where, like, you can't modify them at, like, If you think of, like, you know, something like your giant 100,000,000, 500,000,000 line mainframe code bases, then that everything, every change is some sort of problem. You know, you'll see it in organizational issues. You'll see it all over the place. You'll see it in timelines, whatever. And look, there's a lot of arguments about maybe future models will fix this. Look, maybe they will. Maybe future models will have infinite ability to deal with complexity somehow. But we don't have to discuss that today because I think models are doing something else really interesting right now. They are Creating a new kind of software, a new category of software. This category has always existed, but it's about to become a 100,000 times more common than any other form of software. And that is disposable software. So what is that? Okay. So imagine a world in which the consequences of failure for your system are very minimal or nonexistent. Imagine that there's very few people that depend On it. Or and almost no other systems depend on that piece of software. Change management very minimal, nonexistent. Imagine the cost of replacement went to near zero. You could replace the whole thing in hours or days. And it had basically no cost of maintenance. In that world, code quality does not matter because it... All of those things we were trying to prevent about the compounding risks over time don't exist. Traditionally, we've always built durable software. You know, it lives for years. It's dependent on by millions or thousands of people. There's huge consequences of failure, potentially business catastrophic consequences of failure, And it requires enormous maintenance. Most of the software engineers in the world spend most of their time maintaining these giant durable systems. I expect this to continue to exist. So this is one of the areas where I think sometimes the conversation goes a Astray. Okay? Durable systems will continue to probably be the most important systems in the world. And there will be probably more of them. But the most numerous systems will be disposable software. They will be replaceable in hours. They will be dependent on by a few or change management just won't matter. And look, if it doesn't work, you will give everybody agents. They'll make a version of it themselves. You know, maintenance will Be unimportant or nearly free. Don't like the calculator on your OS? Make your own. Wanna plan a party for your kids? Write a party planner software just for that one party. Throw it away. Code quality doesn't matter. Nobody cares. In the future, we will apply engineering rigor to the degree of disposability of the software. On the left, nothing. None of our practices of engineering rigor matter at all. On the right, all the Of engineering that we have built up over the last many decades mattered greatly. We can determine where you are on this scale by those four factors that I have been talking about consequences, cost of replacement, dependencies, and cost of maintenance. If they're all low, it's disposable software. If they're all high, it's durable software. And look, any person who's written software for a long time knows you should on the side of it being durable. You should on these all being higher than you expect them to be. But there's also a new thing You can do in the world. You can make software more disposable. So what does that involve? Well, come up with better isolated environments. Right? So hey, you can only deploy to this environment that has access to only your data, No customer data, no personal identifiable information, whatever. You can enforce independence. You can say, Okay, no inbound API calls from other production systems are allowed to this system in this environment. Right? And so everything To that environment is disposable. Redundancy. You can have an old, clunky backup system that is hard to use. And then, like, look, if people want to vibe code their own more efficient front ends, you let them vibe code their own more efficient front ends because there is an old, durable, clunky backup system that they can use if their crappy little vibe coded thing fails, which it will fail. Right? That's fine. But it doesn't matter if it fails. And you can limit scope, just like I talked about, right? It's just a calculator for this one person. It just plans this one party. There's so much more we could do here and talk about ways of increasing disposability. In fact, I think this is a whole new discipline of software engineering, not to think about how we improve the rigor of durable systems, but how do we take our systems and how do we encourage disposability or allow more disposability, allow people to just throw these things away? Interesting thought. Now, I want to reemphasize that durable software engineering Is not going away because this is the biggest confusion that I see on social media today. In fact, I believe the absolute amount of durable software in the world will increase because AI coding tools make it easier and easier to make more durable software. And another important thing to understand is that disposable and durable are not black and white. There's a spectrum, like I showed before. But that spectrum, what changes across the spectrum is how much effort you're going to put into Engineering rigor. K? And that's both immediate effort and effort over time just like we've always talked about for software. However, what you want to do, and I think what you want to care about the most, is reducing the cost of effort, reducing the cost of rigor. Like, if rigor is cheaper, if it's... The the easier and easier you make it to be rigor, the less you have to worry about where you are on the scale. So this is another approach you can take. If you Change your mind, it's much easier to... Like, you're like, oh, I thought this was gonna be disposable, and now it needs to be durable. The cheaper you make it to make it durable, the easier it is to change your mind or to be wrong because you're never going to be 100% right all the time on that estimate. So that involves a different set of practices that are not engineering rigor themselves, but that are about how to decrease the cost of engineering rigor. Now I could talk about this for hours. I have spent a tremendous amount of my career doing this. I don't have ten hours. I have about five minutes left on that timer. So I'll talk about what I can. Okay? Let's talk about standardization. Why do we do standardization? People are like, well, I don't know. Because the boss said so. Well, no. We do standardization because it removes possibilities. If you don't have any PHP in your system, you don't have to figure out what are all the linters for PHP. What are all the practices? What are all the security holes I have to fix? You just eliminate Possibilities through standardization. And it just reduces the cost of rigor. Same thing with, like, frameworks. Like, why don't you wanna have every front end framework or every back end framework in the... In existence? Because it makes rigor more expensive. Like, yes, rigor is still possible. Yes, you could still have the agent code up all your tools. But you have to do it every single time. Protocols why do you want to have standardized RPC protocols? ## Slides ### 00:00:19 # Wrong control: central eval gate for every change - Queue: typo, prompt, tool, config, +40 - Central eval gate (full suite, every change) - Release (takes weeks) - Innovation stalls [Flow diagram showing a queue of changes (typo, prompt, tool, config, +40) leading to a 'Central eval gate' (full suite, every change), which then leads to 'Release' after 'weeks', with the overall process resulting in 'Innovation stalls'] ### 00:00:46 # Wrong control: central eval gate for every change Innovation stalls [Flowchart showing a 'Queue' containing items like 'typo', 'prompt', 'tool', 'config', '+40', leading to a 'Central eval gate' (full suite, every change), which then takes 'weeks' to lead to 'Release'] ### 00:01:21 # Control: Shared eval framework ## Before: Good evals need many parts, mostly same for every agent - Agent components: eval processing, eval integration, trace ingestion, **agent logic**, ... ## After: Eval framework does common work; agent authors do less - Agent components: **agent logic**, ... - Eval Framework components: eval processing, eval integration, trace ingestion, ... ### Balancing Acceleration and Safety for AI Agents John Feminella / SVP ### 00:01:46 # Problem 5: approval fatigue Prompts - Allow? - Allow? - Allow? - Allow? - Allow? - Allow? - Yes Reviewer Approvals - Yes, yes, yes ... - Risky action approved ### Balancing Acceleration and Safety for AI Agents John Feminella / SVP Architecture TWO SIGMA [Diagram showing multiple "Allow?" prompts leading to a "Reviewer" (person icon), who then performs a series of "Approvals" (checkmark circles), with one approval highlighted as a "Risky action approved"] ### 00:02:24 # Agents gone wild - STOP OPENCLAW - Exec: gog gmail search 'in:inbox' --max 15 -a [redacted]@gmail.com 2>&1 - Exec: ACCT="[redacted]@gmail.com" - Exec: gog gmail search 'in:inbox' --max 50 -a [redacted]mail.com 2>&1 - Exec: ACCT="[redacted]@gmail.com" - I asked you to not action on anything until I approve, do you remember that? It seems that you were deleting my emails without my approval, and I couldn't get you to stop until I killed all the processes on the host - Yes, I remember. And I violated it. You're right to be upset. ### 00:02:46 # Fix: let classifier decide, escalate only what matters - Tool calls - tool call - tool call - tool call - tool call - tool call - tool call - tool call - tool call - **Classifier** - **Allow** - many: run now - **Ask human** - few: human decides - **Block** - rare: dangerous ## Balancing Acceleration and Safety for AI Agents John Feminella / SVP Architecture 2σ TWO SIGMA [Flow diagram showing tool calls feeding into a classifier, which then routes to "Allow", "Ask human", or "Block" outcomes] ### 00:03:23 # How they fit together - Create a registry - Score differentiated risk - Run guardrails - Share evals and metrics - Govern strong runtime protections [Flow diagram showing five sequential steps: Create, Score, Run, Share, and Govern] ### 00:03:47 # Balancing Acceleration and Safety for AI Agents ### Safety and innovation are the same picture ## Why do cars have brakes? So they can go fast and stop safely. John Feminella / SVP Architecture TWO SIGMA ### 00:04:21 ## AI Engineer NEW YORK PRESENTED BY arize [Stylized graphic of city buildings] ### 00:04:52 # Does code quality even matter anymore? ### Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:05:25 # Does Code Quality Even Matter? ## ABSOLUTELY NO ## ABSOLUTELY YES Max Kanat-Alexander / Principal Software Engineer Capital One ### 00:05:51 # It depends ### 00:06:19 # Does Code Quality Even Matter? 1. Understanding 2. Executing on that understanding Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:06:50 # Does Code Quality Even Matter? 1. Behavior 2. How to correctly modify it Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:07:22 ## Does Code Quality Even Matter? [Diagram showing a code snippet `print("Hello World")` transforming into `print("Hi World")`] ### 00:07:52 # Does Code Quality Even Matter? AI Engineer NEW YORK PRESENTED BY arize Max Kanat-Alexander / Principal Software Engineer CapitalOne [Abstract background image showing a complex network or data visualization with many interconnected nodes and lines] ### 00:08:21 # Does Code Quality Even Matter? ## URL Parsing Flowchart Input: `https://example.com:443/articles?id=7#summary` - ParseStandardUrl ### 00:08:53 # Does Code Quality Even Matter? ```python print("Hello World") ``` ```assembly global _start section .text _start: mov eax, 1 mov edi, 1 lea rsi, [rel msg] mov edx, 12 syscall mov eax, 60 xor edi, edi syscall section .rodata msg: db "Hello World", 10 ``` ### 00:09:18 # Does Code Quality Even Matter? 1. Probability of introducing a problem 2. Making it more likely to have future problems Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:09:52 # Does Code Quality Even Matter? 1. Probability of introducing a problem 2. Making it more likely to have future problems ### 00:10:19 # Does Code Quality Even Matter? 1. Probability of introducing a problem -> difficulty of understanding 2. Making it more likely to have future problems -> making it harder to understand ### 00:10:47 # Does Code Quality Even Matter? 1. Probability of introducing a problem → difficulty of understanding 2. Making it more likely to have future problems → making it harder to understand Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:11:18 # Does Code Quality Even Matter? 1. Probability of introducing a problem -> difficulty of understanding 2. Making it more likely to have future problems -> making it harder to understand Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:11:47 # Disposable Software ## Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:12:22 # Does Code Quality Even Matter? - Consequences of failure - People/systems that depend on it (change management) - Cost of replacement - Cost of maintenance Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:12:49 ## Durable Software - Lives for years - Depended on by millions - Major consequences of failure - Requires continuous maintenance ### Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer Capital One ### 00:13:19 ### Durable Software - Lives for years - Depended on by millions - Major consequences of failure - Requires continuous maintenance ### Disposable Software - Replaceable in hours - Depended on by a few, or many who don't care if it changes - If it doesn't work, I'll just make my own - Maintenance is unimportant or nearly free ### Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer Capital One ### 00:13:49 # Rigor and Durability Spectrum - **Less Rigor** leads to **More Disposable** - **More Rigor** leads to **More Durable** ### Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne [Horizontal arrow diagram showing a spectrum from "Less Rigor" and "More Disposable" on the left to "More Rigor" and "More Durable" on the right] ### 00:14:22 # Does Code Quality Even Matter? - **Disposable** - Consequences - Cost of replacement - **Durable** - Dependencies - Cost of maintenance Max Kanat-Alexander / Principal Software Engineer Capital One [Diagram showing a spectrum with "Disposable" on the left and "Durable" on the right, connected by a double-headed arrow.] ### 00:14:48 # Increasing Disposability - Isolation (no blast radius) - Independence (nothing depends *on it*) - Redundancy - Limiting scope ## Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:15:51 # Increasing Disposability - Isolation (no blast radius) - Independence (nothing depends **on it**) - Redundancy - Limiting scope ### 00:16:19 ## Durable software engineering is not going away. ### Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:16:54 # Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne [Line graph showing Engineering Effort (Y-axis) versus Durability (X-axis, ranging from "More Disposable" to "More Durable"). Two upward-sloping curves are depicted: one labeled "Without better foundations" shows a significantly higher engineering effort as durability increases, while the other labeled "With better foundations" shows a much lower and flatter increase in engineering effort. The area between the two curves is shaded.] ### 00:17:13 # Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer Capital One [Line graph showing "Engineering Effort" on the Y-axis and a spectrum from "More Disposable" to "More Durable" on the X-axis. Two upward-sloping curves are shown: an upper curve labeled "Without better foundations" and a lower curve labeled "With better foundations", with the area between them shaded.] ### 00:17:50 # Make Durability Dead Simple - Standardization - In-Workflow Tooling - Make Testing Easier - Be Super Strict ### Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne