# Fixing the PR Bottleneck — Matt Pocock — session 2026-09-24T14:30:00.000Z → 2026-09-24T15:00:00.000Z _76 transcript lines · 39 slides_ ## Transcript Ladies and Gentlemen, please join me in welcoming to the stage your MC for the AI Engineer Paris 2026, developer relations engineer at Replit, Raouf Shebry. Okay. Okay. All good. How's it going, guys? Hey. This is the last stretch. Okay? So this is why we need to gather all the And we get going for the last stretch of this AI engineer, at Paris. And, I'm so excited about our next speaker. So our next speaker... Actually, you don't need me. He doesn't need any introduction. He was known as the typescript wizard, and then, you know, he was also known for his work at Vercel before he started talking about skills, and then it turns out that he has really strong opinions and good opinions about AI. So our next Here is going to talk to you about how to fix your PI bottlenecks. And my god, I relate to that so much. Anybody relate to that? Seriously. Like, who's who's having, like, so many PRs coming their way and then they they can't review any of them. So, well, without further ado, please join me in welcoming to the stage, Matt Pocock. Alright. Hello. Hello, folks. Having a good conference so far? Having a good conference so far? Okay. Good. So I'm here to talk about fixing the PR bottleneck, and this is kind of a grand title for trying to fix the thing that most organizations struggle with, I think, and have kind of historically struggle with before AI, you know. We have always had huge numbers of PRs just laying around that no one's bothered to review. And this has now increased massively because of the new strains on That's because of AI imposing all these weird constraints. And to do this, I'm gonna use the rubric of my skills, which is you've kinda heard about. Maybe you've used them. And I have a couple of new skills to announce that are going to hopefully improve the way that you do PRs, improve the way that's or improve the speed at which you can review them and do them. Speed. Now we've been pushed to do more with less, essentially, or more PRs, more Work, more stuff. And this is kind of the central promise of AI, that we're going to be able to use these agents to scale ourselves up to do more work. And this has resulted in the software factory, probably the biggest buzzword of the day. Everyone's talking about software factory that I've chat to. And I think of a software factory as primarily something where instead of the human initiating all of this work, we're gonna pass some of that initiation. Some of the initiation is gonna be done by agents. And I think of their You're gonna end up with a slop cannon. Right? You're just gonna end up with a ton of slop crappy PRs that you're not gonna be able to touch or review or even freaking look at. So you need breaks. These are mechanisms that slow down, that increase quality, that make sure that your code base doesn't turn into a software entropy nightmare because code is the environment your agent operates in. And if you have bad code in your code base, that is going to beget more bad code. And so I'm gonna talk about these Three breaks in this talk and talk about how we can use them to counterintuitively go faster. So automated checks. These are the deterministic checks in your repo that we've had for thousands of years or, you know, since the fifties. Deterministic checks where we have linting and tests and type checking and code quality metrics, all of these things going together, and they all work the same every time. Layered on top of that, we have automated review. So we have agents who look it up. A code and say, okay, you know, these are for the things that the tests didn't catch or this is looking at the structure of the code base in general. And then on top of that, the third layer, the final layer is human review. So people looking at the PR. And these three layers form this kind of cake that we end up with when we get to human review. And so more speed, of course, means more PRs. And so the goal here is to make human review faster by leaning on those first two phases. Stop the slop. That's the first principle here, which is if you raise the quality of the code that you're shipping, you're gonna end up doing less human review because it's just gonna be better work, and so you're gonna end up needing to, make for your interventions. So automated checks. Now automated checks are cheap. That's the cool thing about them is that they don't cost tokens like automated review does. They don't cost human effort. They just cost CPU cycles. So these are, for instance, you know, you run your tests on every code chain. Maybe those tests do incur some tokens because, let's say, an agent, creates a bug and the test catch it, then you need to spend some tokens to go and fix it. But those are tokens pretty well spent in my opinion. So checks are cheap. That means you can layer on loads and loads and loads of them on your repos, and you're probably not using enough of them or not being creative enough with your use. But checks can lie. Does a green CI mean that the code is ready for merge? No. It does not. And so we've always needed, on top of these checks, some extra layer to figure out if there's anything catastrophically wrong with the code before we ship it. And so all of the other phases, the human review and automated review, these are lie detectors. These are for finding lies in the automated checks. Now I wanna show you some of these lies, first of all, because this helps when we're thinking about code and thinking about automated checks to see how bad it is and how bad things can get. The first is Tautological tests, a test that just reasserts the implementation. Opus five got addicted to these. I don't quite understand why. It would have, for instance, x post character limit equals 280. Can anyone guess the test that was written to test this behavior? Right? You've probably seen this a thousand times. This is real code from agents or from stuff that I found my agents doing. It said expect x post character limit to be 280. So the implementation looked like that. And the test essentially This is incredible, test. What it's doing here is it's essentially testing whether two things in the UI appear in the right order. So it's checking that the pitch detail page, or rather the video section comes after the content plan. What it does is it doesn't render it to a screen. It just reads the actual file, the module, into its own memory, and then it finds the right thing. So it finds content plan, finds videos, and then it expect The videos to be after it in the source material, which is crazy if you think about it because I can just, like, change the way the source material looks and this test will fail. It's too sensitive to the structure of my code base. So that's another way that automated checks can fail. And there are also oh, sorry. Automated checks can lie. There are also tests that literally cannot fail, and this will feel familiar to you if you've abused mocking in the past or have used various things. For instance here. We have a use audio boost function, and this use audio boost function, its internals use the audio context API in the DOM. Don't worry if you don't know any of this. But what we're doing here is we're just stubbing it out with some fake methods. And it turns out the audio context has some complicated error modes and it will fail if you use it under strange conditions. And so just doing this means our test cannot fail using those modes and we're gonna hit strange errors in production that our tests can't fix. And so the question is then, if you can cheat on automated checks, if, you know and even in good faith ways as well. The the AI isn't trying to write bad tests here. It's just taking our instructions and writing tests that are too tied into the structure instead of actually executing code. So how do we make automated checks harder to cheat? If we can do that, then we can increase the quality of those checks, which means which increases our quality bar. And the first thing I really like about this is code based design. So you can actually design your way out of these bad checks. So what does good code based design look like? I've talked about this before in previous AI engineering talks I've given, which are deep modules. These are modules that hide complex behavior behind simple interfaces. This is a John Osterhaus idea from the philosophy of software design. If we look at these two modules, we've got a, which has a large implementation hiding behind a tiny little Interface at the top. Okay? And then b is a large interface, lots of functions you can call, and those functions individually don't do very much. Does that make sense? Yeah? Now if you have a deep module like a here, you're gonna have fewer structure sensitive tests because you're hiding more of the implementation behind that interface. If it's just testing at that interface, you're gonna get better tests. And so your job here is to force the agent to use that Interface instead of reaching into the implementation to test these weird implementation details. So I've got a skill for this. You can have, like, the weirdest vibe coded, like, code base, the crappiest code base that you've ever set your eyes on, And you can run this skill on it, and it will make it better. What this does essentially gives you opportunities for deepening modules and kinda looks like this. Raise your hands if you've used this skill, by the way. I'm not sure how many of my folks are in this room. Yeah. Okay. It's really freaking nice. Essentially, it gives you a HTML document, I'll get out of the way here, of all of the different, potential opportunities it sees. So we can see here, we have a before and we have an after where we're sort of like reducing duplication. We're creating a nice deep testable module. And then you can go ahead and implement that. And attached to this, there's also this kind of language that I've put together for describing modules because like if you ever try and read up about how to structure a code base, you're gonna Find 20 different approaches and they're all gonna be called DDD. And, like, what you need is a consistent language that you can use in your team to talk about this stuff. And so I have a little code based design skill that defines what locality is, defines what leverage is, defines what a seam is. I was using seams before they were cool. And what locality means is kind of how, well located together all of the code is. How can you change like a small change in one module and have it ripple out? And also leverage is what you get when you have a deep module because the caller, the person who's actually calling that module, gets a lot of value of calling a simple function. Both of those are very good in code bases and good for agents too, it turns out. But I'm sort of describing all of these highfalutin coding standards, but how do we actually make sure the agent does them? Right? How do you make sure the agent creates deep modules and creates good tests and doesn't Write these crap tautological ones or structure sensitive tests. Well, I do think most people get this wrong. And my first piece of advice is don't put coding standards in your implementer agents. Okay? Let me explain this. If you imagine the implementer agent kind of looks like this, where this is all of the things the agent needs to be able to do in its single context window. It needs to be able to explore, like to look For the code that it's going to change. It then needs to actually change it, so make the updates to the files in the green. And then it needs some budget for actually debugging the thing. So for running those, automated checks for actually checking and verifying that it works. Now this is quite a lot of work, it turns out. And if you try to impose your coding standards on it as well, it's going to perform worse. So implementation is overloaded. That's the mental model I want you to have. And so can we find a way to impose those coding standards in a way that isn't so overloaded? Well, this is my effort. This is my code review skill. And it receives a diff, and it reads a file inside the repository called coding standards which you can write, you can customize, and then it checks if the code follows those standards. And so if we look at the reviewer agent here, it also crucially runs it in a sub agent. So it's got its own context window to Have handle here. It's got its own budget. This one, it needs to do some exploration. Right? Because, sure, it receives the diff, so it knows exactly where it's located, where the code is, but it should probably do a bit of exploration just so it has the wider context, understands the code. But it doesn't need to do any implementation, doesn't need to do any debugging. So while implementation is overloaded, Revu is actually underloaded. So it doesn't have that many jobs to do. This means you can pile in a bunch of coding standards to it, and it will Do a much better job than if you try to do it with implement. So I think of this, and this is uncomfortable, right? Because we we all want to be able to just get good code out the first time. But I think of this as the two part process for writing good code, which is implement, you make it work, and then code review, you actually make it good, you impose your coding standards. And for, you know, the retro developers among us, this is essentially a red, green, refactor approach. We use one context window to make it Okay. Do the red green. And then we do another context window to refactor it. That's at least how it works in my head, and it's been very successful for me. This means that when you have coding standards, you don't put them in global scope. You don't put them in agents.md because then they sort of drown out your implementer. It may read them. It may not. You put them in codingstandards.md. And that way, just the code review agent does it. Now I've got another idea here which is, I've been told Lots of people today. Lots of people saying, you know, I've been talking about review and automated review, increase, you know, fixing the PR bottleneck. So many folks say, oh, yeah. We just use a third party service. We use, Cursor Bugbot. We use CodeRabbit or something like that. I think that I've I've really tried to make a generic code review skill in the past that binds all the bugs and does security review and that kind of thing. It turns out it's really, really hard because you either make it too general and it just gives you false Positives that are actually relevant to your use case, or you make it too specific. You say, okay. Find all the TypeScript stuff, and then Rust people can't use it. So I would say don't outsource automated review. Build your own. Build up your own coding standards over time. Share them across your team. And if you have an opportunity to impose coding standards, if you've got some docs sitting around that no one reads, this is the place to put them in. And also, when you have this automated reviewer, a really Natural inclination for lots of people is to say, oh, yeah. My code review agent, what it does is it reads the code and then it comments on the PR. So what your code review agent is doing in that case is it's providing more work for the human reviewer. The human reviewer then has to read all of these verbose comments and figure out, okay, do we implement this, do we implement that? The reviewer should commit. It should make fixes. So it should actually change the things that it finds. Because then when the human comes around, you're review Viewing a really nice artifact. If it finds anything that it has any questions over, then of course, it can comment, but the default should be commits. Stop trying to one shot good code. Stop trying to make the implementer agent the only thing that you do and go, okay, I'm gonna force it to be amazing. It takes a little bit of, you know, thinking your way out of there. But once you realize it, it is fabulous. So okay. With all of that process, we've run our automated checks. We've now run our automated review to make Sure the automated checks aren't lying. How do we then maximize the PR's quality in terms of human review? How do we get it, like, working the best it can? So we need a human friendly PR. And this is a new skill, coming into the repo, which is currently in progress, but I'll be releasing it soon, which is the PR skill. This is one I've been mulling over for a long, long time. I haven't quite figured out what the state of the art is. And I've realized the best way to make a PR skill is just to steal all the best ideas that everyone's And it's very nice. Now what does a good PR body look like? How would you recreate this skill on your own? Well, the first principle is some reviews are more important than others. Not every review is essential. Right? And once you understand this, you realize, okay. That means I can focus my energy on the really important reviews. But how do we categorize them? Well, you gotta think about the PR As using this AWS terminology, which is, is it a one way door or is it a two way door? Now most PRs that you have will be two way doors. You can merge the PR and then always pull it back later. It's the glorious benefit of being a software engineer as opposed to a civil engineer. Right? Mostly when you're a civil engineer, it's a it's a one way door. Right? If you get something wrong, that bridge is going down. But if you have a two way door, it means that you can easily revert the change. Now that might be more, Because our processes now are so sort of streamlined and all we're all collaborating around these same skill files, these same steering files. We're all building an environment for our agents to operate in together. You should think of the process that produces your code as just as important as the code itself. In other words, when you do a human review, you're not just reviewing the code, you're reviewing the system that creates it. And the theory here is that you never want to write the same comment. Twice. Right? You never want to catch the agent doing the same thing over two PRs. And so what's the mechanism by which you can make your human review matter? Well, this is a new skill. This is called retro. Retro for retrospective. You essentially take a session that you've done. It can either be like a single, agent session or it can be a PR plus the session, or you can just get it to look at okay. Look at all of the PRs that we've done over the last week, all of the reviews, pull them all in. Let's do ## Slides ### 00:01:03 # Tuana Çelik DEVELOPER RELATIONS LEAD PRIOR ### 00:01:39 # Sohan Maheshwar ## Developer Advocate Lead Authzed [Authzed logo] ### 00:02:09 # Partners & Sponsors - NVIDIA - Cockroach Labs - Mistral - FACTORY - SENTRY - Sonar - neo4j - Cloudflare - ### 00:02:38 - Astral - AI Engineer - NVIDIA - gfuse by ClickHouse - AI Engineer PARIS - descone [Slide showing various company logos and names] ### 00:03:10 - Notion - Mistral - NVIDIA - langfuse by ClickHouse - AI Engine Paris [Collection of company logos and names] ### 00:03:35 # Matt Pocock ### 00:05:07 # Accelerators Inputs to the software factory - bug reports, support tickets, slow queries ### 00:06:06 ## uality of the software ### 00:06:32 king, code quality. ### 00:07:01 ## Human Review Humans review the PR. The most costly step in the process. ### 00:07:34 # Stop The Slop Higher code quality = faster human review ### 00:08:37 # ...rge? No ### 00:09:05 ```js export const X_POST_CHARACTER_LIMIT = 280; ``` ### 00:09:34 ## Automated Checks # Structure-Sensitive Tests A test that knows too much about the internal workings of the system ### 00:10:06 ```typescript const PITCH_DETAIL_PATH = path.join( __dirname, "..", "routes", "__app.pitches.$pitchId.tsx", ); describe("pitch detail page section order", () => { it("renders Videos section after Content Plan", () => { const content = fs.readFileSync(PITCH_DETAIL_PATH, "utf-8"); const contentPlanIndex = content.indexOf(">Content Plan<"); const videosIndex = content.indexOf('title="Videos"'); expect(contentPlanIndex).toBeGreaterThan(-1); expect(videosIndex).toBeGreaterThan(-1); expect(videosIndex).toBeGreaterThan(contentPlanIndex); }); }); ``` ### 00:10:38 ```typescript const PITCH_DETAIL_PATH = path.join( __dirname, "...", "routes", "__app.pitches.$pitchId.tsx", ); describe("pitch detail page section order", () => { it("renders Videos section after Content Plan", () => { const content = fs.readFileSync(PITCH_DETAIL_PATH, "utf-8"); const contentPlanIndex = content.indexOf(">Content Plan<"); const videosIndex = content.indexOf('title="Videos"'); expect ### 00:11:07 ```javascript // Setup: vi.stubGlobal("AudioContext", () => ({ createMediaElementSource: fakeFn(), createGain: fakeFn(), destination: "mock-destination", })); // Tests: useAudioBoost(video, boostDb); ``` ### 00:11:32 ### Automated Checks ## How do we make automated checks harder to cheat? ### 00:12:01 ## ehind simple interface ### 00:13:00 ...s opportunities to restr... base into deep modules. ### 00:13:37 # ARCHITECTURE REVIEW ## course-video-manager 18 Sep 2026 - scoped to the Learning Goal / Beat / Course View Display Settings / Section Workbench hot spot (last ### 00:14:02 - one interface, three call sites - test passes: deleting the module today changes - three components with a source of truth ### 00:14:37 **Problem:** the same compound visibility expression is hand-rolled at three call sites with no shared name, and none of them is unit tested — `section-card.tsx` has no test file ### 00:15:01 ## Automated Review # Don't Put Coding Standards In Your Implementer ### 00:15:37 # Automated Review **Implementer** - Exploration - Implementation - Debugging [Horizontal flow diagram showing five rectangular boxes. An arrow from "Exploration" points to the second box. ### 00:16:06 # Automated Review ## `/code-review` Receives a diff, reads `CODING_STANDARDS.md`, and checks if the code follows the standards. ### 00:16:34 # Exploration [Yellow curved line above the text] ### 00:17:07 ## Automated Review `/implement` = Make It Work `/code-review` = Make It Good ### 00:17:35 # CODING STANDARDS ### 00:18:06 ## Automated Review # Don't Outsource Automated Review ### 00:18:34 ### Automated Review ## Don't Outsource Automated Review ### 00:19:01 comments to review and ### 00:19:34 # Stop Trying To One-Shot Good Code ## Automated Review ### 00:20:05 code quality, concrete crete evidence. ### 00:20:33 ## Human Review # Some Reviews Are More Important Than Others ### 00:22:07 # Merge Danger - Door: Two-way — trivial to revert, changes only how an error message is rendered, no behavior change to the turn loop itself. - Blast Radius: Localized — only affects the text of error messages surfaced from stream "error" parts (auth/bridge/compact failures) across all commands, not just `/compact`. No API, schema, or control-flow changes. ### 00:22:36 ## Human Review # /show-me From the humanlayer/skills repo ### 00:23:06 ### Human Review ## Make understanding the "why" as fast as possible ### 00:24:33 ### Human Review ## /retro Suggest automated checks & coding standards to make the next review easier.