# What we learned by analyzing 1M AI-generated PRs — Daksh Gupta — session 2026-06-30T19:05:00.000Z → 2026-06-30T19:25:00.000Z

_72 transcript lines · 0 slides · source: full recording_

## Transcript

So we engineered memory on purpose in layers. Working memory for the last few minutes, pinned facts about someone who is. Episodes, whole conversations as little stories, relationships with warmth that grows from stranger to trusted, and a bouncer at the door, a salience kit that decides what's even worth remembering So the brain doesn't fill up. Invuity without making things up. Hey. My name is Daksh. I am one of the co founders of a company called Greptile. And at Greptile, we are working on AI agents that review and test pull requests with full context of the codebase. The way we do this is when you open a pull request, Graptile reads all the files, it uses a knowledge base of the codebase to understand what could potentially be affected, it also runs your code in a sandbox to find various issues. But I don't want to talk about graph tile today. What I wanna talk about is something very interesting. So when I first moved to San Francisco three years ago was when AI coding had really just started to work. And what we had at that time was Copilot with Tab Complete and later Cursor with more advanced Tab Complete. In 2024, AI agents got a little bit better and we started to see multi file edits. This was with Cursor and early versions of hard code. In 2025, something interesting started to happen, which is the agents got increasingly autonomous. And December 2025 was the turning point for anyone that's working in AI coding or using AI to code. Understand that those models were fundamentally different from the ones that came before and that they were completely autonomous. They could go from ticket to pull request. And from there, Twitter seemed to be full of stories of people experimenting with polyphasic sleep to keep their agents alive and building large amounts of code all at once. And to me, it seemed like this was something that was possible inside the realm of maybe indie hacker projects or startups. Not in the enterprise, which was all of the companies that we were working with. And this notion that I had that truly agentic coding could not take off the enterprise was starting to get challenged because I heard many reports of people doing this stuff in the enterprise and that it was really working. And so, I got really curious, are these fully vibed PRs actually good? Can you do this at a large scale company with real customers? And in what ways did they fail when they did fail? So as an amateur data scientist, I decided to dive into the data and see what came of it. First thing I discovered was it's actually very hard to figure out which PRs are fully Vibe Coded. I've got to probably work with tens of thousands of teams, and we have this interesting place where we're not the coders, but only the reviewers of the code. But all this code goes through our system at any of these companies, companies like NVIDIA and Coinbase and Scale, real companies with real customers and critical workflows. Identifying which PRs were vibe coded was difficult, but I decided to give it a shot. And the first thing that I tried was I looked for the GitHub author field. Every commit in GitHub has to have a field for the author, and I figured that if the author was attributed to codecs or cloud code, that was pretty strong evidence that that was a fully vied PR. But I found, of all the PRs and again, more than a million pull requests go through GraphQL every month only about one out of every 100 were attributed to an agent in the author field. Intuitively, this didn't make sense to me. It couldn't be that only 1% of all PRs were fully Vibe coded. And so I started looking for PR description footers. Turns out, these agents are as narcissistic as humans are and they like to leave traces that they are the ones that worked on the code. And So Claude would leave a co authored by Claude, and codex would leave a co authored by codex, and that seemed to help a little bit. The third thing I looked at was branch prefixes. Codex actually names the branches for you, and it seems to me intuitively that if someone was using codecs to produce the branch names, there was pretty good evidence that most of that PR was Vibe Coded. I found that in the month of April, more than a quarter of all PRs that Poll reviewed had evidence of being, if not fully, at least largely vibe coded. Then I tracked this number back to the previous twelve months, and it turns out that the rate of change is incredibly fast. About a year ago, less than a percent of all poll requests had evidence of being fully vibe coded. And in April, that number was more than a quarter and trending upwards. And then became the real question, are these PRs any good? Are real companies doing this, are they any good? Can you merge this stuff? Is this just slop or is it real, good software? And of course then, the question becomes, what is Mean for a pull request to be good? What does it mean for a Vibe coded or a human written pull request to be good? So then I started to do some statistical analysis around PR quality. And the first thing that I looked at was revert rates. GitHub lets you revert pull requests and it seemed reasonable that if a a pull request was reverted, it must have not been very good. And so, I started tracking reverts across our entire customer base. Conveniently, GitHub labels reverted branches with revert dash Under the name of the PR. And so I started looking at the rates at which pull requests were being reverted. This is the data that I found. With codecs, about one out of every thousand pull requests seem to be reverted. For Devon, that number was three and a half. And for humans, interestingly, for PRs where there was reasonable evidence that it was written largely by a person or at least owned by a person, that number was around two and a half. These numbers are fairly within range. They're all small, less than a percent, and all kind of within range of each So I started looking at some other pieces of evidence because I was very convinced that these Vibe Coder PRs aren't good. So I was really looking for evidence that that was true. So the next thing I looked at was, are the reverberates different by size? Because it seemed that it was possible that people were making agents do simpler work. It was the straightforward, well defined tickets that agents were working on, and humans were taking on the more complex tasks, and that's why the revert rates were similar. And it seemed like PR size was a good proxy for complexity of a change. But it turns out that revert rates actually stayed relatively stable across different sizes. There wasn't even a real pattern that said that agents were working on smaller For simpler tasks. It turns out revert rates were pretty even across the board for agents and for humans. The next thing I looked at was greptile comments. If greptile's the reviewing and testing and validating agent for all of this code, it seemed reasonable that it would find more issues in code that was worse. And Graph Tile conveniently labels all the issues with p zeros, p ones, or p twos. And so I started tracking the rates at which Graph Tile finds p zeros, p ones. Than p twos across pull requests that are written by people versus ones written entirely by AI. Once again, there just was not that much of a difference between the human written and AI written pull requests. In fact, humans were actually more likely to make p zero errors than dev in codex or Claude. Similarly, the case with p ones where both dev in codex created fewer p ones and p twos were even closer. Once again, there just wasn't that much evidence that the agent written PRs were meaningfully worse or more critically damaged than their human Counterparts. The next thing I looked at was review rounds. A very common pattern for using reptile is that you have it review a pull request, give it some comments, and then you let the agent fix those comments, make another commit in the pull request, and usually, two rounds is enough for all of the issues from the pull request to be addressed and for the pull request to be ready to be merged. It seemed reasonable that a smarter, more capable author for a pull request would be able to get to a mergeable PR with fewer review rounds that seemed intuitively true to me. And so I started tracking average number of review rounds. With Devon, a little over two. For people, 2.2. For codex, two and a half. Once again, entirely within range. The mean number of iterations it took for an agent or a person to get a pull request to a mergeable state was fairly consistent throughout. Once again, no evidence of agentic PRs being worse. So now that there wasn't any strong evidence that these were quantitatively worse or better, I was interested in seeing if there were patterns of failure that were different in agent written PRs and human written PRs. Were they bad in different And so I started tracking some interesting data around that. Here's how I did it. So, grubtile leaves comments on all of these pull requests. It tends to identify all kinds of logic performance and security errors. And so I decided to run a very simple text search and said, okay, are types of errors in different agents different for different sites? Now, the dataset here is quite large. It's in the millions of pull requests. And so this data is Reasonably high confidence. And of course, you can't use old data, so I only used about two months of data since prior to that, the agents are essentially obsolete. And I saw some very interesting data points. Claude, for instance, is one and a half times more likely than people to produce a SQL injection error. Error. Auth bypasses are half as likely coming from Devon than from any other agent and from people. And another interesting data point but n plus one query errors were much more common in cursors than any other agent, far more common. And so it's interesting to me that there were this much variation in the qualitative aspects of what These agents were getting wrong, when there was so much consistency in how good or bad they were. It had been measured in many different ways. So it might turn out that autonomous agents are actually quite good. And this is very exciting to me because, to an extent, I'd kinda become a programmer before AI had taken off and I was used to, like many of us, the cruft of programming. And it seemed like a lot of the vibe Coded agents that we were seeing in 2024 and 2025 were a lot more promise than substance, especially when applied to professional commercial settings. But the trajectory seems quite clear. These agents are getting really, really good and they're getting really good at coding, not only in general environments. Grappheld's customer base is made up much more of larger companies than it is of individual coders. Of course, individual coders don't really review their code or don't need to. And so this sample set is essentially created from companies with real customers. And it turns out that they actually can do a lot of True vibe coding in a commercial setting. So we started thinking about what that means for code review and code validation. If most of the code is going to be written entirely by agents and it seems like the agents are writing pretty good code, then what should code review look like? I think a couple of things are very clear. Interesting data point that emerged from my statistical analysis is that there was quite a bit of a difference between the ninety ninth percentile of coders and the fiftieth percentile of coders in terms of how much code they're writing. In fact, the median code That uses grptile is writing fewer than 50 commits a month. The ninety ninth percentile is writing close to a thousand commits a month. At that scale, it is completely infeasible, especially this is going to be the standard for how people code to do anything resembling manual code review. It also seems not very intuitive that we would be maintaining large test suites. As a grub tile, we got really excited about this idea that we could build truly autonomous code validation. We started thinking about what that means and of course, code validation in the prior era was made Of these three general facets. You'd have testing, you'd have review, and you have quality assurance. But to us, these seem like implementation details. The goal of code validation was actually just to answer three very simple questions. Does this change violate the user contract? Two, does this change increase the future propensity of a user contract violation? And third, does it fulfill what the author intended? And so we've designed WebTile from the ground up to do exactly that, to answer those three questions. We do this by reviewing the code, of course, in a more traditional sense where we use agents to look at the code, look at the blast radius of the change, files outside of the diff that could be affected or And then we run the code in a sandbox. Click around with web browser agents to try to break things. Simulate users, in a sense. That has turned out to be a much more effective path towards truly autonomous code validation. Because it's hard for me to imagine that this idea that humans would be reviewing large amounts of agent code and that would become the new Java software engineer is quite dystopian. I don't think I wanna wake up every morning and review Slop all day. I'd much rather spend the time coming up with the genius ideas and letting machines figure out how That's my talk. My name is Daksh, and you can try GRAPTIL today. You should also come visit us at our booth. We have an arcade machine. Thank you so much. Hi. I'm Amol, CEO of Nouri Agentic. We deploy an AI employee that understands your company, your code, docs, Slack, and other kinds of data, we spend a lot of time thinking about how coding agents really work. Most people think coding agents only write code. But if you ask Hey, that's just bad marketing. Forget the name for a second. Coding agents can do almost anything. There's just one trick. You have to be able to think like an agent to get it to do what you want it to do. Today, we're gonna talk about how we use coding agents to do something most people think agents are terrible at. Make visual artifacts. Like slides, docs, and yeah, even video. Every day, the world pours something like thirty four thousand human years into making slide decks. Most of that time Isn't the thinking, it's the fiddling. A deck that takes ten hours should really take about twenty five minutes once you remove all the formatting and the branding and the moving things around. Say you need to make a slide. What do you do? You open a tool, PowerPoint, slides, Figma, Canva, and then you start manipulating a Canvas. Every one of these tools is built for human hands and Human eyes. Click, drag, drop, resize, snap to grid. All motions and patterns that make sense for our geospatial view of the world. There is a data structure underneath, but it's in a format that only the application can read. What happens when you hand these tools to an agent? Well, the output comes out all wrong. Things overlap in weird ways. You can't see the text. There's no alignment. It's just garbage. AI skeptics say that it's not just the tools. Agents fundamentally can't reason about space. And there are whole benchmarks like arc GI that are built exactly around that premise. There's a famous little test for this from developer Simon Willison. He asks every new model the same thing. Can you draw a pelican riding a bicycle? But there's a trick. The agent is only allowed to use SVG. It's a quick gut check for whether a model can reason about space at all. Here's some examples of what the models actually give you on this. Test. And yeah. These are pretty bad. Like, genuinely, deeply, really bad. So does that mean it's hopeless? Agents are just doomed to be bad at graphics? No. I don't think so. If you ask me, it's not the model, it's the medium. If I asked you, someone who is presumably human, to hand write an SVG of a pelican, you wouldn't be able to do that either. SVGs are just a wall of numbers. You can't go from a wall Numbers to a pelican, you just can't see that way. That's just not how people think. We think graphically. So we build tools that let us draw on a canvas. Figma MCPs, PowerPoint CLIs, screenshot and replace loops. What do all of these agent tools have in common? They all approach the problem like a human. But an a I is not a human. Asking an AI to use a canvas is like asking a human to write SVG by hand. It doesn't really make sense. You need to give the AI tools based on how it thinks, not in pixels, in Language. Words, tokens, structure, that is its native medium. Imagine a language that's incredible at describing layout, that models have seen and trained on billions of examples of, that they understand intuitively, that renders to pixels and can run everywhere. Oh, right. HTML lets a model think in structure. HTML tags have meanings built into the language. A heading, a chart, a grid, and the browser turns it all into pixels. So the model never actually places a coordinate, and you can get all sorts of visual effects, charts, and layouts, fonts, and motion, all of it for free. Remember that pelican from earlier? Now ask it to do the same exact task, but in HTML. Same bird but now it's in a structure that the model can reason about. And you can read and theme and edit every single line of it. I spent my whole life building slide decks. PowerPoint. So I always thought that those two things, slide decks and PowerPoint, were synonyms. But that's just not really true, is it? PowerPoint is a tool that you use to make slide decks. The deck itself, that's just the presentation mode. And as it turns out, no one in your audience is gonna care how you got to the presentation mode. The editing format is totally arbitrary. So you can just Pick the editing format that the agents are already good at, HTML, and if you need to, render to a different format like PDF later on. We use this HTML trick to build all of our slide decks, our board decks, and our sales decks. These are real things that we actually present and send out constantly. We use it for our docs too. It gives our docs color and vibrancy, all while following our brand. And, of course, we also use it to make videos, like this one. What you're watching is just HTML and CSS. It's literally just divs all the way Down. Almost everything is better with a little structure and a little bit of color. Plain text is a choice, generally a choice of convenience, but it's usually the wrong one if you're actually trying to create something of use. Now, I do wanna take a quick beat here and point out that a beautiful deck on its own is generally not worth anything. You still have to go and get all of that Content, all of the things that actually populate that deck. Right? Well, again, we can think like the model. If you just give the model access to your data, say your call transcripts or your emails, you can have the model build the deck end to end. Let your agents do all the grunt work while you focus on vision and story. That's what Nori sessions lets you do. I've built entire board decks for my phone on the subway during my commute. Why? Because our NoriBot lives in the fabric of our company. Of course, Nori ships with everything you need to make this all work. So, don't bother reinventing the wheel. That's my little spiel. Thanks for listening. If you have just one takeaway, it's this. Stop thinking like a user. Think like the model. Give it the right language, and for graphics, all you need is HTML. Hi everyone. 10 x. You feel it yet? Hi, my name is Aurel Zainen and I'm a mobile software engineer for the last fourteen years, and I'm here to talk to you today about 10 x, reimagining the mobile dev workflow. So you know, back in the old times when cursor was that thing you make with your mouse, and, AI agents were that dystopian character from sci fi, books, or movies, whatever fits your style. You know, just a few months back. Back then when we thought that we will still be using our IDEs, just maybe slightly

## Slides