Coming possible today. I think it is good to step back a little bit and just look how fast this has happened. And so I, you know, I graph this out and really this is kind of, you know, look at GPT 3.5, and, you know, that was the first time I really experienced, what an LLM was, and it was, you know, a mind blowing experience. But if you look how quickly this capability is then being compressed into models with fewer and fewer parameters, up into the present Day, you know, this can now fit in a billion parameters roundabout and then run on a smartphone or run on a Raspberry Pi in your hand. This is phenomenal. And I by the way, I know that benchmarks are not perfect, and then the LU probably less so. This indicative of a trend, which is gonna continue. You know, if I'm here next year, this is gonna have gone down even further. And it's it's not just general knowledge question answering, it's reasoning too. And so here we're looking at a frontier of, you know, what can you do with 4,000,000,000 active parameters? The kind of compute on, today's edge devices. And again, you know, this is gone, from especially with mixture of experts coming into the picture, gone from something which is now, exceeding what, Claude Sonnet, 3.5 was doing when it was first released with its Frontier GPQA Diamond score, I think this was like a year and a half ago. This stuff is just moving phenomenally fast. And, of course, it is not just about capability. It's about Latency when we're talking about human user interaction. And there are also models now which can run comfortably on device, which do, speech to text in a latency, which are useful for creating voice UI without any need to send this inference to the Cloud. And I think it's very interesting to consider what drives, you know, these latency requirements. And it's really, you know, it's what people expect. What they're comfortable with is the way we should be designing these things. I Find this graph really fascinating. So this is an anthropological study of human conversation turn taking and the latency between speakers. And it's interesting that it's culturally dependent and language dependent. So Japanese is very quick and efficient in responding. Danish is very laid back and very slow. I don't know if that's because the verb is at the end of the sentence, by the way. And then English is kind of in the middle. And so that gives you a kind of gauge of of where these things need to land. And so, yeah, so I hope you are as As as I am about all of this inference being possible on edge devices, you may say, well, you know, we could do this on the data center already. Why, you know, why put it at the edge? And it's it's a good question. I think there's some, you know, very, very good reasons for this. Number one is efficiency. And so, you know, I think many people are already looking at their token usage and thinking, how how much of this could I offload onto a local device or a device on prem? There's the fact that you can customize this. So maybe you Want to use personal data or proprietary data to your organization and you would like to keep this, in house. And then there's kind of this opportunity for a tent sorry, intention aware devices where maybe they're observing your speech or have cameras, and are constantly looking and listening. And a lot of people may want to choose whether that stuff is shared off the device onto a data center or not. So I think these are some really compelling reasons why we would want to put Inference at the edge. And so I did say, you know, there's something else going on here. And I think it's this combination of you've got this accessible hardware. You've got these open weight models. Putting this together, I think, is is a fascinating opportunity. And so I don't know how many people are like hardware makers or into hardware here. The hardware people here can have a show of hands. Good. Okay. I'm glad to see that. And so, you know, it's been notable that for many years Now people have been building pretty amazing, hardware innovations using these composable modules from, like, Raspberry Pi or Arduino, maybe in the first instance, and then, creating totally new designs based on this stuff. And I think that combination of hardware innovation plus these open models is gonna open up some really interesting potential. I think the observant among you may notice that the micro duck is in the corner there. And, you know, my my favorite example of this combination, because I don't think it's really As widely, you know, it's not happening yet as widely as it could do. But a really nice early example of this is the hugging face robotics. Oh, sorry. Hugging face pollen robotics, Ricci mini. And So I know Ricci mini is from France. I was very happy to be on the beta, test the program for Ricci mini. So last Christmas with my son, I built Ricci mini, and I was really happy to see. So inside of this thing, it has a Stewart platform so it can express emotions with its head, and it has got a bunch of Those that, move the head platform. So this has ARM Cortex seven zero in there. They have actuators on the antenna. They have a Raspberry Pi compute module four for the Wi Fi version. And so with that, you can either run some applications on the device itself or you can use that to then call inference services off device. And so that's super cool. But it's not just hardware. It's also an SDK, which is open source, and it's a community of developers. Creating really cool applications for this platform. And here is one application example I would like to highlight. So this is by our very own Marco Domingo at Arm. And he created an app which basically, it sits on your desk. And if you use your phone, like start doom scrolling or getting distracted, it tells you not to. And this this won a prize. So this won him a VIP ticket to NVIDIA GTC in San Jose this year and a DGX Spark. And it was my Ricci Mini. I have to say, I'm not bitter about that. But well done, Marco. It's a cool app. And actually, if you go on the App Store, it is there today on the it's a hugging face space where they keep Ricci Mini Apps. So in the design of these types of on device applications, you know, there's a engineering challenge because you have constraints. Right? So you have to think about the available computer and memory. And oftentimes, you will have to divide up this task to make best use of the available hardware. So for a vision app. Location is quite common to maybe have, like, a motion gate. Maybe this triggers object detection. And if you classify things you're interested in, you grab a region of interest. Maybe you do image embeddings, and maybe only then do you go and then fire off a BLM. And so these tasks may be carried out by different types of processor on device and in some cases may then go off to other devices entirely or even to the data center to be run. And here's another example. So this is the app. Made during the beta test of Riichi mini. And this is running entirely on a Raspberry Pi five. What I love about this is you, you know, you say something and Riichi will immediately turn his head. And there's something magical about embodied AI in the way that, you know, a smart speaker may just bong or the LED lights up. But having a character turn its head to face you and then respond is a really cool experience. And, I think in doing this, you know, I was using a Raspberry Pi, which is definitely capable of running a bunch of different LLMs, but just because of the Latency constraint. I chose to use embeddings and then do a semantic match and call tools against this. So you could still ask it the weather or the time, and it could generate using text to speech on device also and reply. And here's Marco's demo. And by the way, I forgot to say, so in the main expo outside after this talk, Marco will be showing this, demo live, and you can ask him questions about it. Fundamentally, what we have here is he has his DGX Spark, so he's got a more capable device. From his prize wedding. And he's running, another interactive agent like demo. This time, he's using minstrel minstrel three on device, and he's doing speech to text. And then he's creating responses, synthesizing a response using text to speech and playing it back. In addition to that, he's doing a bunch of tool calling. And so he has this ability to you know, you can ask it what it is saying. It will Tell you, you know, actually, maybe I'll play the video. That could be a better way of doing this. Let me just do that really quickly. Okay. Yes? Can you greet the room? Hi, everyone. I'm Rishi. Nice to meet you all. I've got a badge here for AI engineer, and let's see if Rishi can do anything with it. Hey, Rishi. Yes? Can you meet the attendee? Show me your badge. Nice. Got it. Give me a second. Hi, Dominic. Good to see you at the AI Engineer Conference. I see you're from Arm. They're known for designing risk based CPU architectures powering everything from smartphones to supercomputers. Now finally, can you dance? How was that? Too much? Wants me to dance again? See you later, everyone. Ritchie's off to recharge. Have a great day. Alright. Hey, Rij. So I love the character that, and personality that Riji has through, minstrel, by the way. Too Much is a really cool, demo. If you go outside, like I said, you can actually go and see if it can read your badge and tell your fortune, etcetera. It is really cool to actually interact with it in person. So those are a couple of examples, but I did talk at the beginning about intent. And so you probably heard about the intention economy and you know, the attention Is, you know, you get people looking at apps and there's adverts and and stuff like that. The intention economy is understanding, what people want and providing solutions to it without them having to spell out everything. I think that's the way I would describe it. And here's a little example of an interaction that could be possible through this intent. So, you know, if you have, for example, reaches a bed light bed night lamp there, that's also connected to your home assistant system, for example. If you ask it, you know, you tell it you're done for the day, it may not be able to interpret that right. If it understands your preferences in terms of, you know, AC temperature or lighting, it can go and act on it. If it's got ambient context, and if Ricci is maybe looking or listening and remembering, has memory, about the context it's it's receiving. It can maybe see, oh, your partner's asleep. And so maybe Riichi doesn't say anything. It just nods and it kind of dims the lights, and it goes to sleep itself. And I think, you know, this ability to deliver a better UX with fewer interactions with a device is really the key to this thing. This is just one small example, and I think there are thousands of others. And inherent in this is the fact that, you know, there there is an agent running here. So we talked about vision and audio and sensor streams and maybe integration with with Home Assistant. These are effectively events or maybe state that can be queried from an agentic system. And of course, you know, with with the agent, there's a harness which is running on a CPU, and that CPU is doing a bunch of work, managing that context and and the memory. And then, I think the, key thing here is how much of the inference can be done off or off of the device. And I think this is like an orchestration problem, which a lot of the industry are focusing on today. Certainly, you know, task aware routing and being able to judge whether a task is possible to run on a device is is really key. And we'll look at that in a second. And finally, you know, we've been talking a lot about consumer based applications and how people interact with devices. There are a ton of embedded systems out there that have computers that actually, know, maybe they're in industry or smart greenhouses or, you know, all kinds of commercial building systems where, they're mainly monitoring sensors or maybe doing actuation, but with limited, interaction from a person. And so I really like to think about this idea of of those thousands of embedded systems that are out there. Is there a potential for, you know, an agentic embedded system? So I am not saying that the agent carries out all of these tasks, more that the agent takes the role of a supervisor and when things happen out of band, in other words, you know, if there's a fault that the designer of the system hadn't anticipated, can these agents improvise to go and fix them? And the kind of thought experiment, I think is really interesting is Bits per second communication to do this. And so, you know, it's interesting to consider, could agents be used to preserve these systems and monitor them, and keep them in good health? And I think this is one of the really interesting avenues for exploration. On the tip on the topic of hybrid AI, so, you know, there are a ton of interesting research papers in this area and there's some open source projects, you know, Light LLM or Root LLM. That you can apply to this challenge of, you know, can inference run on this constrained device in a model that fits there? Do I need to send it externally to a more capable piece of hardware? And so I was running some experiments the other day that, you know, there's a couple of different strategies. I think that the embedding kernel is the the quickest because you don't actually have to run the prompt. You really just, you know, you have some kind of calibration set. You look at prompts that successfully ran locally and ones that didn't, and you see which one is more similar to your If a prompting question, there's other solutions out there like calibrated uncertainty or even decoder state probing, which are possible once you've actually run inference. You've already drafted a response to the prompt, and it may be then you need to kind of defer this off device to be run, if the confidence is low or if it classifies as something which should would have failed. This is super interesting stuff. I have to say, you know, these evals are really just looking at single, single prompt. Eval sets. There are, you know, there's pinch bench, Clory. Is it Clory? Now there's a bunch of eval's looking at multi step tasks. I tend to think, you know, it's really your own application, which is the best thing to consider in these cases. So I've talked a lot about what we can do with a combination of really efficient models and accessible hardware. And just like the inspiration of creating devices, they're a little bit more, I think, you know, a more more Considered unless obtrusive. I think this is something that would benefit everyone. But, of course, you know, I should also talk about the hardware beneath a lot of this stuff. And I mentioned at the beginning, I work for Arm. And Arm has processor solutions that span from these really low power, wearable or, you know, sensor node type devices with our Cortex M and Ethos, up to Linux based platforms with Cortex a, which you might find in, in Jetson or in Raspberry Pi. Right the way up to data center infrastructure, and our focus is providing a common software framework and, we have recently announced an AI portal which has optimized models and also performance optimization tools. And one thing, you know, I would definitely encourage you, you know, I'll show this QR code at the end. Please register, become an early access customer to this thing. We're interested to get your feedback and improve this for the applications you'd like to develop. And, Just a couple of examples. So, you know, we've upstreamed kernels for Clidy AI into LAMA CPP into Onyx runtime. So when Gemma four came out, it ran optimally on SME two. But it's not only the new stuff. There's also consideration for, really widely used kind of established models and hardware. So I really like this example, which we have just released or due to release very shortly, which is Ultralytics YOLO 26, kind of opt Specifically for the Raspberry Pi five, ISA. And so they got 1.87 times faster inference. I think the key thing to point out here is, you know, typically, when you look at the FPS numbers for some of these models, they're occupying if they're running on CPU, all four cores. And really, in an application that is not ideal, you would like some headroom to do other stuff. And this kind of optimization is the difference between, you know, having two cores free to run your application, run other Kinds of algorithm on that output. So this is really interesting stuff, I think. Okay. So if you would like to get more resources, there's that QR code again for developer.arm.com/ai. Please sign up. Yeah. There's early access to our optimization tools, which I would encourage you to get into. Also, I'm really happy to connect with you either on LinkedIn or I'll be out by Ricci later on. And so, yeah, just to say again. So Marco Domingo, I think he'll be there for PM. So not directly after this, but at the break. He'll be on the the expo stage with Ricci Minnie if you'd like to go and chat with him and interact with Ricci. Alright. Thank you very much. Cheers. So I feel connected to to Cognition. How you feeling, everybody? Yeah. Good? Yeah? Okay. You know what I'm gonna do. Right? How are you feeling, guys? Okay. I think the next one is gonna be great. Yeah? Well, thank you so much, thank you so much, Dominic, for the presentation. Now we're gonna move on to Our next speaker who's going to talk to us about agentic map reduce for large scale coding agents. I'm super curious about this one. I wanna see what what this means. But please join me in welcoming to the stage VP of Engineering, EMEA at Cognition, Yanis Sorakis. Okay. Let's see. Oops. Am I connecting to the right projector? Classic. Please give me a few seconds to sort it out. Uh-huh. That's good. That's good. It's promising. And I need to see how I extend my screen. Slush. Excellent. Thank you very much for having here. It's such a great pleasure to be speaking to AI engineering in Paris to this great audience. Very excited about what I have to show to you today. An approach on tackling very Arge a codebase wide problems. And, yeah, let's get going. A few words of intro for us. We're cognition. We're in and apply AI research lab. Our founding DNA is in, AI research, competitive programming, competitive maths. We're based in San Francisco, but, we are now all around the world, including London. This is our European base of operations where I'm based. Across the most, like, sophisticated organizations in the world. They range from, systemically critical banks, Goldman City, Santander all the way to the US Army. Big tech, smaller, like a fast moving, tech native startups. And, we love tackling like a really, really like difficult, coding tasks. And I want to reflect on our experience. Rinse's on sort of like the classes of problems that one, can see. Let us consider let's start by think two categories. Okay. Let's call them kind of like localized, agentic workloads, which are basically kind of like, take the your median session with an agent. I want to implement a feature or given a particular area of the code base. I want to kind of like a write some tests or augment the existing test coverage. Given a particular, like, slice of my DB proxy, I want to optimize queries and so on and so forth. This concerns work. It's kind of like a like a bounded problem. It is like a set of relevant files in the code base. Okay. That basically modularize the whole logic. And equally importantly, if I were to tell an agent what, what to do, I will either point them. Explicit to the files. Or if I don't do, they can trivially go and find the relevant area neighborhood of the code base to carry out the task. Now let us consider what I would kind of like simply call codebase wide task. And by the way, codebase wide doesn't only mean like a single repo. You can have multiple repose. Let's take, for example, a case that we're going to revisit again again today. Security scanning. What is the security profile of my application? We know that there are, like, the individual security kind of like a quality levels of different modules of the code base. But what is equally important is the different kind of like the ways that interplay and how those things can evolve in the real world, how they can connect in order to reveal vulnerabilities similar in a similar vein, extracting common logic. We work with clients that have, like, really, really large code bases. They want to commonize ways they treat arithmetic, financial formulas, or even kind of like infrastructure ways of interacting with the database. They're all sort of like, first of all, it is a search problem, but it's also a problem of sort of like understanding and finding all the different nuances. Okay, by which you need to a common library needs to be extra. Acted need to conform to and so on and so forth. There's a complex interplay between the components. Take the security example, the security quality of my individual, let's say microservices. Okay. Could be kind of like a, you know, okay. I have like, if I were to look at them in isolation, low, priority vulnerabilities, they can be the backlog, but when chained together, they can give rise to a critical exploit. And then, therefore, like for those tasks, the result is trustworthy when you consider the totality of the code base or the totality of the deployed application for sets of applications. Alright. And, of course, one can say, kind of like, yeah, that's why we have n to n tests. Okay. That's why kind of like in our SDLC, we have a different quality gates and so on and so forth. Absolutely. Our goal is always called to for to ship. If left, right, I I do to catch those things and be aware of those things as, as soon as possible. Now if if I were to double click a little bit on sort of like, what the challenges, an agent could face in those code base white tasks. I'm telling, can you please find? I want to find instances of a particular formula of a particular logic that appears in different variations across my code base because I want to extract to a common library. Okay, there is a challenge of finding the work in the first place. Okay, it is a search problem. Okay, agents will spend a lot of time searching and backtracking like burning tokens. Okay, number two, polluting the context. There is a great great amount is going to build up. It's going to be competing for attention very, very variable, in diverse type of context. And number three, what is effectively are stopping criteria? Okay. Again, to make this a strong argument as possible, one will say, Yeah, of course, we're not gonna solve those big problems. Okay, in a kind of like in a single session, there will be some notion of agent orchestration. Okay? There will be some notion of, okay, we're gonna have a kind of like a, an the agent agent outer loop. We're gonna break down the problem into different tasks. Okay? We're gonna define I need dug how those things need to be completed. And then we're going to have some notion of inner loop on the agent. We have to build all that stuff. Okay? Because obviously normal, nobody is gonna one shot prompt. Hey, kind of like, please fix the security vulnerabilities. Absolutely. My argument is today is to present to you a framework. Okay? How we think about the problem that gives the best, both in terms of outcome performance and cost. And we make This, okay, as a first class notion in our product. I touched upon this before, kind of like single agent reasoning. I'm not gonna label too much on this point. Okay? Both in terms of kind of like time spent searching, context, pollution, and so on and so forth. Single agent problem does not solve very, very large code base wide tasks. How we approach this? Okay. So, we propose, okay, a variation. Or an or an augmentation of the, map reduce framework, an idea from, distributed systems. How many people here have worked with, map reduce? Awesome. So super, super quick example. Mapreduce introduced by Google was a way to solve to perform analysis on like a really, really large data sets that they are distributed, that they live on a distributed file system. Okay, so you Oh, so the classic example that we give, hey, we have a huge textual corpus on the distributed file system, different files, and we need to do some lexicographic analysis. I'll keep it simple. Count the number of words on each book. Okay, they will be a series of map functions. These are simple stateless as in pure functions that they kind of like would count each book. They can run-in parallel, and there's a reducer step that Consolidates the results in a very similar vein with adaptation. We want to port this into the agentic coding world. Okay, where instead of like huge data sets, consider huge code bases. Okay, an overview of how this thing would work. Okay. Firstly, there are two steps that we introduced at the beginning. There is a main agent that creates a plan. For investigation number three, for each of those candidate areas, we spin a series of parallel sub agents where they look these areas and only these areas, and they perform in the analysis. This is the map step. And then there is a reduced step. The main agent, again, considers the results, the combined results, the distilled results from all the different sub agents, okay, and performs Some reconciliation, some consolidation analysis. This this can be duplication, prioritization, or some more richer analysis on the combined result. Let's kind of like double click a bit on those Concepts. Okay. And I'm going to use, I'm going to use security. Okay. I was going to like a motivating example. Okay. I have this large code base. And what is the security kind of like, the security quality security? Level of this code base. Okay, I will provide the threat model. I will provide any additional context from the code base. Okay, what the pre work step is going to do is okay. We're gonna write does determine a deterministic search function that will identify areas of the code base of interest in the context of security. You folks can imagine API boundaries, where tokens are being minted and be passed Found, SQL and, code entry points and so on and so forth. The outcome of that is to produce basically those deterministic selectors they're gonna run, and they're gonna produce like a series of batches. Okay, or a series of shards of the code base where we can then run each one of those sub agents. Okay, those are pages. Then they have a very nice clean and bonded context, which they can perform the investigations and reports. Build these are the most critical, and there is a ranking of them. Number three, like this is crucial again in the context of codebase wide tasks, presence of chained vulnerabilities. Remember, individual agents may tell you that their individual narrow part of the code base has some low, critical defining. But what the reducer step can do is can assess combinations of those low vulnerability, low criticality finding. Can they produce something much more serious? And again, with the right harness, okay? Devin can run. Okay. Can actually test. Okay. And try to simulate those attack paths. And to do it kind of like a little bit more visually, we treat the tend to treat a code basis as basically a graph. Okay? All there's all those references between, the different parts of the code base. Selectors look for off areas, look for API entry points, data SQL entry points. And then from those selectors that they have run cheaply and deterministic on the code base, we deploy, okay, individual devins, individual agents that perform security analysis for this particular narrow bounded path of the code base. And then on the reduced step, Devon will produce something like this. Okay, where it will sort of like number one triage number two, the duplicate number. Three run and test and try to recreate those vulnerabilities, including kind of like false positives as well. So, yeah, mention about reproducing because obviously, at the end of the day, you don't want to ship kind of like tickets because you we know whenever we have a release, you know, we run security vulnerability from tools that are like tons and tons of things to churn through. We want to be shipping and like legit PRs against a known vulnerability. So recreating is extremely, extremely important. Now we're pushing the PRS and just want to comment. Okay, a little bit on kind of like the relationship between cost and performance. Okay, of course, kind of like you can run very kind of like expensive agents. You can run them against really large code bases. Okay, what is extremely important, as I mentioned earlier, is to think hard and have the best possible. Ways, okay, to mitigate the cost of searching. Searching is a huge problem in large code basis and context pollution and context overflow. This is solved by them. Devon and the main agent are in the agentic MapReduce, writing those deterministic selectors and creating lots and lots of banded context, which end up being much cheaper and end up having a better recall profile rather than, compared to competing models. Okay. And we've run this one fifty on the data sets 50 vulnerabilities from kind of like a GitHub, and they run across like a multitude of languages, Golang, Python, Java, and so on and so forth. Another thing that we should think about, I talked a lot about security, agentic map reduce in the context of vulnerability detection and remediation. Think about it about maybe it can be a generalized concept across all different Code base wide workloads. Okay. Dead code removal. I mentioned a lot extracting common common logic into some centralized library from a very large code base, dead code detection and so on and so forth. Okay? And this is something that again, we from our research, we have kind of like moved it to security and we elevating it into a first class feature, in our product. We are, as I mentioned, kind of like earlier. We are, applied ai research lab. We were doing work on areas like agentic map reduce building our own coding models. So two point o recent to model harnesses like dev infusion, agentic infrastructure. Please check out our research blog. And if you have any questions, this is my email. Looking forward to your comments and see you around in. The conference. Thank you very much. Thank you so much, Yanis. Let's give it up for Yanis once more, please. Alright. Okay. So we all deserve a break right now. I know that, it's been a long day for everybody, but we have a great line of of speakers still. Okay? So we're gonna be back here at 04:30, please, and, enjoy your break. Alright. See you in a Ladies and Gentlemen, please join me in welcoming to the stage your MC for the AI Engineer Paris 2026, developer relations engineer at Replit, Raouf Shebry. Okay. Okay. All good. How's it going, guys? Hey. This is the last stretch. Okay? So this is why we need to gather all the And we get going for the last stretch of this AI engineer, at Paris. And, I'm so excited about our next speaker. So our next speaker... Actually, you don't need me. He doesn't need any introduction. He was known as the typescript wizard, and then, you know, he was also known for his work at Vercel before he started talking about skills, and then it turns out that he has really strong opinions and good opinions about AI. So our next Here is going to talk to you about how to fix your PI bottlenecks. And my god, I relate to that so much. Anybody relate to that? Seriously. Like, who's who's having, like, so many PRs coming their way and then they they can't review any of them. So, well, without further ado, please join me in welcoming to the stage, Matt Pocock. Alright. Hello. Hello, folks. Having a good conference so far? Having a good conference so far? Okay. Good. So I'm here to talk about fixing the PR bottleneck, and this is kind of a grand title for trying to fix the thing that most organizations struggle with, I think, and have kind of historically struggle with before AI, you know. We have always had huge numbers of PRs just laying around that no one's bothered to review. And this has now increased massively because of the new strains on That's because of AI imposing all these weird constraints. And to do this, I'm gonna use the rubric of my skills, which is you've kinda heard about. Maybe you've used them. And I have a couple of new skills to announce that are going to hopefully improve the way that you do PRs, improve the way that's or improve the speed at which you can review them and do them. Speed. Now we've been pushed to do more with less, essentially, or more PRs, more Work, more stuff. And this is kind of the central promise of AI, that we're going to be able to use these agents to scale ourselves up to do more work. And this has resulted in the software factory, probably the biggest buzzword of the day. Everyone's talking about software factory that I've chat to. And I think of a software factory as primarily something where instead of the human initiating all of this work, we're gonna pass some of that initiation. Some of the initiation is gonna be done by agents. And I think of their You're gonna end up with a slop cannon. Right? You're just gonna end up with a ton of slop crappy PRs that you're not gonna be able to touch or review or even freaking look at. So you need breaks. These are mechanisms that slow down, that increase quality, that make sure that your code base doesn't turn into a software entropy nightmare because code is the environment your agent operates in. And if you have bad code in your code base, that is going to beget more bad code. And so I'm gonna talk about these Three breaks in this talk and talk about how we can use them to counterintuitively go faster. So automated checks. These are the deterministic checks in your repo that we've had for thousands of years or, you know, since the fifties. Deterministic checks where we have linting and tests and type checking and code quality metrics, all of these things going together, and they all work the same every time. Layered on top of that, we have automated review. So we have agents who look it up. A code and say, okay, you know, these are for the things that the tests didn't catch or this is looking at the structure of the code base in general. And then on top of that, the third layer, the final layer is human review. So people looking at the PR. And these three layers form this kind of cake that we end up with when we get to human review. And so more speed, of course, means more PRs. And so the goal here is to make human review faster by leaning on those first two phases. Stop the slop. That's the first principle here, which is if you raise the quality of the code that you're shipping, you're gonna end up doing less human review because it's just gonna be better work, and so you're gonna end up needing to, make for your interventions. So automated checks. Now automated checks are cheap. That's the cool thing about them is that they don't cost tokens like automated review does. They don't cost human effort. They just cost CPU cycles. So these are, for instance, you know, you run your tests on every code chain. Maybe those tests do incur some tokens because, let's say, an agent, creates a bug and the test catch it, then you need to spend some tokens to go and fix it. But those are tokens pretty well spent in my opinion. So checks are cheap. That means you can layer on loads and loads and loads of them on your repos, and you're probably not using enough of them or not being creative enough with your use. But checks can lie. Does a green CI mean that the code is ready for merge? No. It does not. And so we've always needed, on top of these checks, some extra layer to figure out if there's anything catastrophically wrong with the code before we ship it. And so all of the other phases, the human review and automated review, these are lie detectors. These are for finding lies in the automated checks. Now I wanna show you some of these lies, first of all, because this helps when we're thinking about code and thinking about automated checks to see how bad it is and how bad things can get. The first is Tautological tests, a test that just reasserts the implementation. Opus five got addicted to these. I don't quite understand why. It would have, for instance, x post character limit equals 280. Can anyone guess the test that was written to test this behavior? Right? You've probably seen this a thousand times. This is real code from agents or from stuff that I found my agents doing. It said expect x post character limit to be 280. So the implementation looked like that. And the test essentially This is incredible, test. What it's doing here is it's essentially testing whether two things in the UI appear in the right order. So it's checking that the pitch detail page, or rather the video section comes after the content plan. What it does is it doesn't render it to a screen. It just reads the actual file, the module, into its own memory, and then it finds the right thing. So it finds content plan, finds videos, and then it expect The videos to be after it in the source material, which is crazy if you think about it because I can just, like, change the way the source material looks and this test will fail. It's too sensitive to the structure of my code base. So that's another way that automated checks can fail. And there are also oh, sorry. Automated checks can lie. There are also tests that literally cannot fail, and this will feel familiar to you if you've abused mocking in the past or have used various things. For instance here. We have a use audio boost function, and this use audio boost function, its internals use the audio context API in the DOM. Don't worry if you don't know any of this. But what we're doing here is we're just stubbing it out with some fake methods. And it turns out the audio context has some complicated error modes and it will fail if you use it under strange conditions. And so just doing this means our test cannot fail using those modes and we're gonna hit strange errors in production that our tests can't fix. And so the question is then, if you can cheat on automated checks, if, you know and even in good faith ways as well. The the AI isn't trying to write bad tests here. It's just taking our instructions and writing tests that are too tied into the structure instead of actually executing code. So how do we make automated checks harder to cheat? If we can do that, then we can increase the quality of those checks, which means which increases our quality bar. And the first thing I really like about this is code based design. So you can actually design your way out of these bad checks. So what does good code based design look like? I've talked about this before in previous AI engineering talks I've given, which are deep modules. These are modules that hide complex behavior behind simple interfaces. This is a John Osterhaus idea from the philosophy of software design. If we look at these two modules, we've got a, which has a large implementation hiding behind a tiny little Interface at the top. Okay? And then b is a large interface, lots of functions you can call, and those functions individually don't do very much. Does that make sense? Yeah? Now if you have a deep module like a here, you're gonna have fewer structure sensitive tests because you're hiding more of the implementation behind that interface. If it's just testing at that interface, you're gonna get better tests. And so your job here is to force the agent to use that Interface instead of reaching into the implementation to test these weird implementation details. So I've got a skill for this. You can have, like, the weirdest vibe coded, like, code base, the crappiest code base that you've ever set your eyes on, And you can run this skill on it, and it will make it better. What this does essentially gives you opportunities for deepening modules and kinda looks like this. Raise your hands if you've used this skill, by the way. I'm not sure how many of my folks are in this room. Yeah. Okay. It's really freaking nice. Essentially, it gives you a HTML document, I'll get out of the way here, of all of the different, potential opportunities it sees. So we can see here, we have a before and we have an after where we're sort of like reducing duplication. We're creating a nice deep testable module. And then you can go ahead and implement that. And attached to this, there's also this kind of language that I've put together for describing modules because like if you ever try and read up about how to structure a code base, you're gonna Find 20 different approaches and they're all gonna be called DDD. And, like, what you need is a consistent language that you can use in your team to talk about this stuff. And so I have a little code based design skill that defines what locality is, defines what leverage is, defines what a seam is. I was using seams before they were cool. And what locality means is kind of how, well located together all of the code is. How can you change like a small change in one module and have it ripple out? And also leverage is what you get when you have a deep module because the caller, the person who's actually calling that module, gets a lot of value of calling a simple function. Both of those are very good in code bases and good for agents too, it turns out. But I'm sort of describing all of these highfalutin coding standards, but how do we actually make sure the agent does them? Right? How do you make sure the agent creates deep modules and creates good tests and doesn't Write these crap tautological ones or structure sensitive tests. Well, I do think most people get this wrong. And my first piece of advice is don't put coding standards in your implementer agents. Okay? Let me explain this. If you imagine the implementer agent kind of looks like this, where this is all of the things the agent needs to be able to do in its single context window. It needs to be able to explore, like to look For the code that it's going to change. It then needs to actually change it, so make the updates to the files in the green. And then it needs some budget for actually debugging the thing. So for running those, automated checks for actually checking and verifying that it works. Now this is quite a lot of work, it turns out. And if you try to impose your coding standards on it as well, it's going to perform worse. So implementation is overloaded. That's the mental model I want you to have. And so can we find a way to impose those coding standards in a way that isn't so overloaded? Well, this is my effort. This is my code review skill. And it receives a diff, and it reads a file inside the repository called coding standards which you can write, you can customize, and then it checks if the code follows those standards. And so if we look at the reviewer agent here, it also crucially runs it in a sub agent. So it's got its own context window to Have handle here. It's got its own budget. This one, it needs to do some exploration. Right? Because, sure, it receives the diff, so it knows exactly where it's located, where the code is, but it should probably do a bit of exploration just so it has the wider context, understands the code. But it doesn't need to do any implementation, doesn't need to do any debugging. So while implementation is overloaded, Revu is actually underloaded. So it doesn't have that many jobs to do. This means you can pile in a bunch of coding standards to it, and it will Do a much better job than if you try to do it with implement. So I think of this, and this is uncomfortable, right? Because we we all want to be able to just get good code out the first time. But I think of this as the two part process for writing good code, which is implement, you make it work, and then code review, you actually make it good, you impose your coding standards. And for, you know, the retro developers among us, this is essentially a red, green, refactor approach. We use one context window to make it Okay. Do the red green. And then we do another context window to refactor it. That's at least how it works in my head, and it's been very successful for me. This means that when you have coding standards, you don't put them in global scope. You don't put them in agents.md because then they sort of drown out your implementer. It may read them. It may not. You put them in codingstandards.md. And that way, just the code review agent does it. Now I've got another idea here which is, I've been told Lots of people today. Lots of people saying, you know, I've been talking about review and automated review, increase, you know, fixing the PR bottleneck. So many folks say, oh, yeah. We just use a third party service. We use, Cursor Bugbot. We use CodeRabbit or something like that. I think that I've I've really tried to make a generic code review skill in the past that binds all the bugs and does security review and that kind of thing. It turns out it's really, really hard because you either make it too general and it just gives you false Positives that are actually relevant to your use case, or you make it too specific. You say, okay. Find all the TypeScript stuff, and then Rust people can't use it. So I would say don't outsource automated review. Build your own. Build up your own coding standards over time. Share them across your team. And if you have an opportunity to impose coding standards, if you've got some docs sitting around that no one reads, this is the place to put them in. And also, when you have this automated reviewer, a really Natural inclination for lots of people is to say, oh, yeah. My code review agent, what it does is it reads the code and then it comments on the PR. So what your code review agent is doing in that case is it's providing more work for the human reviewer. The human reviewer then has to read all of these verbose comments and figure out, okay, do we implement this, do we implement that? The reviewer should commit. It should make fixes. So it should actually change the things that it finds. Because then when the human comes around, you're review Viewing a really nice artifact. If it finds anything that it has any questions over, then of course, it can comment, but the default should be commits. Stop trying to one shot good code. Stop trying to make the implementer agent the only thing that you do and go, okay, I'm gonna force it to be amazing. It takes a little bit of, you know, thinking your way out of there. But once you realize it, it is fabulous. So okay. With all of that process, we've run our automated checks. We've now run our automated review to make Sure the automated checks aren't lying. How do we then maximize the PR's quality in terms of human review? How do we get it, like, working the best it can? So we need a human friendly PR. And this is a new skill, coming into the repo, which is currently in progress, but I'll be releasing it soon, which is the PR skill. This is one I've been mulling over for a long, long time. I haven't quite figured out what the state of the art is. And I've realized the best way to make a PR skill is just to steal all the best ideas that everyone's And it's very nice. Now what does a good PR body look like? How would you recreate this skill on your own? Well, the first principle is some reviews are more important than others. Not every review is essential. Right? And once you understand this, you realize, okay. That means I can focus my energy on the really important reviews. But how do we categorize them? Well, you gotta think about the PR As using this AWS terminology, which is, is it a one way door or is it a two way door? Now most PRs that you have will be two way doors. You can merge the PR and then always pull it back later. It's the glorious benefit of being a software engineer as opposed to a civil engineer. Right? Mostly when you're a civil engineer, it's a it's a one way door. Right? If you get something wrong, that bridge is going down. But if you have a two way door, it means that you can easily revert the change. Now that might be more, Because our processes now are so sort of streamlined and all we're all collaborating around these same skill files, these same steering files. We're all building an environment for our agents to operate in together. You should think of the process that produces your code as just as important as the code itself. In other words, when you do a human review, you're not just reviewing the code, you're reviewing the system that creates it. And the theory here is that you never want to write the same comment. Twice. Right? You never want to catch the agent doing the same thing over two PRs. And so what's the mechanism by which you can make your human review matter? Well, this is a new skill. This is called retro. Retro for retrospective. You essentially take a session that you've done. It can either be like a single, agent session or it can be a PR plus the session, or you can just get it to look at okay. Look at all of the PRs that we've done over the last week, all of the reviews, pull them all in. Let's do