# A New Age of Software Engineering — Walden Yan — session 2026-11-12T19:20:00.000Z → 2026-11-12T19:40:00.000Z _47 transcript lines · 38 slides_ ## Transcript Alright. Hello, everyone. It's a pleasure to be here today. Quick introduction. I'm Walden. I'm one of the cofounders and chief product officer at Cognition. We build a AI product called Devon. I wanna spend most of my time here today though kind of abstracting a bit away from that and and focusing on what are we doing internally as an engineering team to move way faster. And so The backdrop I would put for all this is this is what our merged PR throughput looks like at Cognition. The start of this is around twelve months ago, so you're looking at about a snapshot of a year. And over this last year, this blue chart here, this 40x, is a 40x in the number of merged PRs we are producing on our engineering team. Now you might also notice there's a small black line along the bottom there. It's easy to miss. But that 2.7x, that is our engineering head count growth in the same Period. So we have been hiring engineers, but each of our engineers is now more than 10 x more productive than they were before. And I would dare to say that these PRs are just as high quality, if not higher quality, than they were a year ago. And so really the crux of what I want to talk about today is what did we do to unlock this 40x engineering throughput? And there's a lot of things we did from both the code base to the way we structure engineering teams that I think Are very practical steps you can take if you want to reproduce the same results. Many of you are probably familiar with the software factory. How many of you guys have heard of software factory? Let's see a raise of hands. Cool. So I'll start with the basics and then I'll work my way up to the more advanced stuff. The basic idea of a soccer factory is, yeah. By default, everything should be self driving. We have PRs that go from initial bug report to final merge and monitoring and maintenance without a single human involved. Oftentimes, human judgment is needed though. That's where we have escalations. But by default, our entire software development process is just run by Devon from end to end. One of the things this means is that the role of the software engineer has to change quite a bit. Around March of this year, we had this conversation with the engineering team. And we said, look, if you are right now just picking off tickets off a linear board, and then sending that to an agent and implementing it, you're not going to really be set up for success in the age of AI. Because anything that is described Can be implemented, will be able to send to Devon. So really, everyone has to change the way that they're developing. And the way we've enforced this is we've taken our larger teams and we've broken them up into smaller pods. And these smaller pods not only are given swarms of agents to do work for them, they are given more vertical responsibility. So instead, you might have only two or three engineers on a team at a time, but they're responsible for everything beyond the engineering as well. They're responsible for listening to customer bug reports. They're responsible for doing QA testing. They're responsible for doing the PMing and product management. And basically every engineer manages their entire own linear board of things to do. And they send that to an agent to implement for them. So it's really a fundamental change in the way that our engineers do work. This has been one of the things that we've implemented over the last year and we've been helping a lot of large organizations implement for themselves as well. Another thing I'd really like to highlight is we work from Slack everywhere. So this is an example of what Customer bug report might look like. They send me a video on Slack, and we just forward bug reports straight into our internal channel. And we tag Devon and say, hey, please fix this. Devon immediately comes up, shows up at PR, but then also sends back a video with the fix and proof. And this makes our lives so much easier, because I literally don't have to leave Slack for most of my work. We've abstracted away all the development to be based on evidence, video. Those. And I can just send the proof that the fix is ready right back to the customer without having to open an IDE or even GitHub most of the time. So this is kind of the end goal you want to reach, which is a point where you can do your entire development life cycle without opening up an IDE or without opening up a single coding agent. But to get to this point, there's a lot of infrastructure we had to change underneath. And so this is the hard part that really every engineering team To do if they want to get to that point. It's not just you purchase a coding agent. You've got to go change the way you build. So before, when we test our applications, we'd be testing them against cloud deployments. We'd have a staging environment, connect to a staging database, connect to live servers, and then we'd have our engineers manually go click through and verify that the flows work end to end. We did a big migration to move every service to be able to run locally with doc and compose. We've made it so every third party service, we have mock versions to be able to run locally. And these can all run inside A single virtual machine that every devon has access to. And every code agent in our company now does not need to touch any live services at all to test their changes. In fact, these VMs, we rebuild them every hour at the company. Every hour we pull the latest version of the code, we rebuild the entire code base from scratch, should be able to run very quickly and easily from scratch. And if these builds ever fail because your code base is rapidly changing, we also have Devon be the first response there. Always immediately fixing these VM snapshots. So every time we spin up an agent from Slack, it goes to a fresh machine on a fresh version of the code base with the latest updates. And by the way, these agents, you need to teach them how to test your code base as well. But the benefit of using cloud coding agents is that they can self maintain their training loops. So that last section there on testing, Devon will write his own skills for how to test our code base. The code base will change over time, so the way you test it will change. But Devon Can keep on updating these skills so that it tests it over and over again. So the end result of this is literally people do not need to open their laptops anymore to test any code changes. In fact, several engineers at Cognition have already moved to using iPads for their development process. By the way, iPads are also more secure. You know, you've seen a lot of these hacks recently. A lot of it is because of dev ops and people leaking keys. Who knows what's on your developer machine? You might have secrets all over the place. When everything is in the cloud in these controlled VMs, they're actually in a much more secure spot. I've showed some basic versions of things you can test. We've also tested advanced multiplayer setups, which is really cool. We've oftentimes, when we were developing Devin's ability to join voice in Zoom calls, we actually had Devin, when it tests, create multiple accounts, have them join the same Zoom call all from within their one VM, and send voice memos back and forth to make sure that the functionality is working. So you can do quite advanced setups if you really push the limits of your code base. And actually, on Top of all this, you know, you obviously want recursive self improvement. I think people think about this from the model sense quite a lot. But actually, if you think about it, every single line of code that's being written, every single PR that's being generated, every decision, if you have a cloud agent set up, that's all tracked across your entire code base. So there's no reason why you should also be able to run agents on top of that, do the data analysis, and figure out where are we wasting time, where are we wasting tokens, what things can be turned into scripts, what tests can we be adding. To improve this process. And so we have a channel where dev everyday posts ideas about, hey, if we change our node set up, we can shave like a minute off of our CI. If you change this lint script, we can like catch 20% more of these bugs before they ever hit AI code review. And this has actually been a huge unlock where you can just set up these systems and the agents will go and optimize your software factory for you. This has been really exciting. One of the things that I don't hear people talking about But I think it's really where things are going next is moving from the world where you're giving tasks to your agents to giving jobs to your agents. And this is a bit of a paradigm shift. You really have to start treating these things like coworkers. So what does giving a job to an agent look like? Imagine I care about back end performance. Really this is what the development flow of the future should look like. You should send to your agent, hey, I want to improve p 99 latency of my application. Instead of you telling it what to do to do that, It should say, Okay, p 99 latency. Let me add the telemetry to figure out where our p 99 latency is today. Let me set up alerts so I can figure out where we need to be in the future. Let me go break down every single task we need to do to be able to move this latency. Let me create data dog alerts. Let me create weekly reports in Slack. And let me send you updates on your phone every day on here's where p 99 latency is and here's how much it's improved. This is fundamentally the type of systems that we encourage all of our engineers to be building at cognition. We don't even want you to be prompting your agents most of the time. We want you to just be giving your agent goals that they can work toward over long durations of time. And anything in your company that you think isn't measurable, find a way to measure it. You know, do you care about your application performance? Go measure the latency of clicks on your application. If your application has a lot of weird layout shifts, go measure the bounding boxes and try to figure out how you can minimize layout shifts across your application. There's also p and nine latency, but there's also things like onboarding Retention. How much of what you guys do as engineers is things you can boil down to a single number or metric? And I encourage you that if you can do that, you should just set up an agent to go optimize that number 20 fourseven. So a great example of this is inference latency. At Cognition, we're lucky to be the first receivers of NVIDIA's new class of Vera Rubin GPUs. Kernels were not built for these GPU's when we received them. So we had to, from scratch, figure out how do we build our own inference stack on top This new hardware that's coming out. And agents have allowed us to do this in a way that is much more efficient than before. So we just tell it, hey, we care about the throughput, our TPM, of these GPU's. Go figure out how to push that to its limits. And over the course of what is about, I guess here, like a week, the agent was able to go. We allocated it a permanent set of GPUs to work on. It was able to go find different strategies, test them. The strategies that it found worked Test. It was sent to a human engineer for final approval. But basically there's eight or nine different things that were roughly discovered via just giving these agents these high level goals. And we were able to push the throughput on these new and Vera Rubin GPU's from what was worse than the new GB three hundred's to what is now significantly better than the previous class of GB 300 GPU's. And again, this is all done by just giving a end goal to your agents. Another example that I like to share with folks. We are a fairly small engineering team. It means we do not have, oftentimes, the bandwidth to fully staff things that we might want to do, like an Android application. But the other day, one of our engineers on our team, Samuil, he had this brilliant idea. He said, Okay, look, we have a team of people working on building a great iOS application. Can't we just use that as a blueprint for our Android application and tell Devon, hey, go maintain parity from iOS application to our Android application end to end. And that is exactly what he did. But it wasn't just, hey, one Devon session to go do all this. He said, set up permanent hooks against our code base to monitor the iOS application. You know, go listen to the Slack channels for user feedback about our applications. And then go recursively improve our application. Go do live QA testing of every detail and making sure it all works. Syncing these things live has been really cool. And I think the final results here are actually quite sick. So we really just gave our agent this one high level prompt, which is, hey, go manage the entire Android application. And over the course of forty four days, we have a work application, which we'll be putting out soon. But as a time lapse, most days there's not a single human message. Going to this agent working on it. And on those days, you know, it's merging upward of like 40, 50 PRs a day. All of this is being live QA tested against the iOS application, pixel for pixel, testing whether escalating only the most important decisions to a human at the end of the day. And most of the time our engineers don't even need to open dev in. They're just from Slack looking at the recordings that it sends back to them. So in the end, the human is barely doing anything in this loop. This is how many messages a human actually had to send to our agent in this entire process. Yeah. And it's not a slop application. This is actually something that we will be going live with very soon. So cool. That's actually all I had. I think the end message from all this Is if you set up the loops for agents correctly, you can do some pretty incredible stuff. So I think really most teams are limited by the amount of ambition they have. So it's time to be ambitious. Thank you. ## Slides ### 00:00:02 [Dark background with a subtle halftone pattern of grey dots forming a central, slightly lighter, curved shape.] ### 00:00:37 # AI Engineer Europe 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. ## ExCeL London ## 02.17 - 02.19 2027 3 DAYS - LONDON, UK [ai.engineer/europe](ai.engineer/europe) [Black and white photo of the London Eye] [Black and white photo of Big Ben and the Houses of Parliament] [Black and white photo of a grand building on a London street] ### 00:01:07 # A new era of software ### 00:01:36 # Cognition's delivery capacity is exploding [Bar chart showing "Merged PR volume" (blue bars) increasing from 1x in Nov 2025 to 40.4x in Oct 2026, while "Engineering headcount (R&D)" (black dots on a dotted line) increases from 1x to 2.7x over the same period.] ## A New Age of Software Engineering Walden Yan / Chief Product Officer Cognition [QR code for live translation ### 00:02:08 # Cognition's delivery capacity is exploding ### A New Age of Software Engineering Walden Yan / Chief Product Officer ⚙️ Cognition [Bar chart showing Merged PR volume (blue ### 00:02:41 # Default autonomous; Human judgement where needed - **Triage**: Organize and prioritize requests - `expect broken` - `list is slow` - **Build ### 00:03:09 # We've restructured our teams to ship more ## Before - Organized by Role - PRODUCT: PM - ENGINEERING: ENG, ENG, ENG, ENG - QA & ON ### 00:03:41 # We've restructured our teams to ship more ## Before - Organized by Role - **PRODUCT**: 1 PM - **ENGINEERING**: 4 ENG - **QA & ONCALL ### 00:04:05 # We've restructured our teams to ship more ## Before: Organized by Role - **Product**: PM - **Engineering**: ENG, ENG, ENG, ENG - ** ### 00:04:36 # Debugging, Testing & Merging all from Slack ## Instant bug report turnaround 1. Customer reports a bug 2. Forwarded to the internal channel 3. Devin replies with a PR and proof 4. Shared back to the customer ## A New Age of Software Engineering Walden Yan / Chief Product Officer ⚙️ Cognition [Screenshot of a Slack conversation in `#team-listenlabs-cognition` reporting a bug about a missing comment in code.] [Screenshot of a Slack conversation in `#internal-bug-reports` discussing a bug where file-level comments are offscreen.] [https://cognition-labs.com/](https://cognition-labs.com/) ### 00:05:09 # Debugging, Testing & Merging all from Slack ## Instant bug report turnaround - Customer reports a bug - Forwarded to the internal channel - Devin replies with a PR and proof ### 00:05:37 # Building an Agent-Testable Codebase ## Before Testing against live services ## After Everything on one VM ## Hourly VM snapshots Devins repair failed builds ## Testing skills ### 00:06:09 # Building an Agent-Testable Codebase ## Before Testing against live services ## After Everything on one VM ## Hourly VM snapshots Devins repair failed builds ## Testing ### 00:06:41 # Building an Agent-Testable Codebase ### Before Testing against live services ### After Everything on one VM ### Hourly VM snapshots Devins repair failed builds ### Testing skills Devins fix and maintain them ## A New Age of Software Engineering Walden Yan / Chief Product Officer Cognition [https://cognition-labs.com/](https://cognition-labs.com/) [Diagram showing a "Before" state with an application interacting with multiple cloud services and SaaS platforms, transforming to an "After" state where the application and its dependencies are containerized within a single VM.] [Timeline diagram showing hourly VM snapshots, with a failed snapshot at 11:00 triggering a repair process.] [Flowchart illustrating a testing skill: a `SKILL.md` file triggers a run, which can fail, and an agent can then fix the failure.] ### 00:07:09 # Building an Agent-Testable Codebase ## Before Testing against live services ## After Everything on one VM ## Hourly VM snapshots Devins repair failed builds ## Testing skills ### 00:07:41 # Agent Testing of Multi-Player Setups ## One spoken phrase, speaker Mac to Devin ### Speaker Mac VM: Devin Speaker (Mac) ``` $ say -v Daniel -a " ### 00:08:09 # Recursive Self-Improvement for Agents - **PAST SESSIONS ANALYZED**: 1,284 - **WHAT DEVIN FOUND AND CHANGED** - **EVERY CHANGE FLOWS BACK INTO THE NEXT SESSIONS** [Diagram illustrating a process where 1,284 past sessions (represented by small rectangles) are fed into an "Analyze" node, and the resulting changes are fed back into subsequent sessions in a continuous loop.] ### 00:08:37 # Recursive Self-Improvement for Agents ### Past Sessions Analyzed 1,312 ### What Devin Found and Changed **Repeated Failure** - One flaky e2e suite retried 1 ### 00:09:09 # We Give Agents Jobs, not Tasks ## A New Age of Software Engineering Presented by Cognition Walden Yan / Chief Product Officer [https://cognition-labs.com/](https://cognition-labs.com/) ### 00:09:37 # From delegating tasks to delegating outcomes ### Human: express a desire - Walden - Make our P99 endpoint latency faster. - Started Devin ### Devin defines goals and ### 00:10:09 # From delegating tasks to delegating outcomes ## Human: express a desire - Walden - Make our P99 endpoint latency faster. - Started Devin ## Devin defines goals and ### 00:10:41 # From delegating tasks to delegating outcomes ## Human: express a desire - Walden: Make our P99 endpoint latency faster. - Started Devin ## Devin defines goals and ### 00:11:04 ## Example 1: 2.8x TPM on NVIDIA Vera Rubins OUTPUT TOKENS/MIN PR GPU (FC1.1 AGENTIC WORKLOAD, 8-GPU PA ### 00:11:37 # Example 1: 2.8x TPM on NVIDIA Vera Rubins OUTPUT TOKENS/MIN PR GPU (FC1.1 AGENTIC WORKLOAD, 8- ### 00:12:09 # Example 2: Android App From Scratch - iOS - Android - Auto-sync iOS Features - Monitors Slack channel for bug reports ## A New Age of Software Engineering ### 00:12:37 # Example 2: Android App From Scratch - iOS - Android - Auto-sync iOS Features - Monitors Slack channel for bug reports ### A New Age of Software Engineering ### 00:13:09 # A New Age of Software Engineering Walden Yan / Chief Product Officer ⚙️ Cognition ⚙️ Cognition [https://cognition-labs.com/](https:// ### 00:13:37 # A New Age of Software Engineering ## Day 34 Nov 1, 2026 ### Pull Requests So Far 1,064 ### Mer ### 00:14:09 # What it took from Samyok - **183** messages - **1,961** words - **22** days without a single message > "this is all just devin" > "be more ambitious" > "make sure all the sessions are ultra" Six pull requests merged for every message he sent. ### 00:14:41 # It's time to be ambitious. ## A New Age of Software Engineering Walden Yan / Chief Product Officer Cognition [AI Engineer Code Summit logo] [Google DeepMind logo ### 00:15:05 # AI Engineer Europe 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 ### 00:16:07 # Code Summit ## AI Engineer ### 00:16:36 ### AI Engineer Code Summit Presented by Google DeepMind ## LIVE TRANSLATION // AVAILABLE Scan the QR code to access real-time translation. [https://deepmind.google/events/code-summit/live-translation/](https://deepmind.google/events/code-summit/live-translation/) [QR code for live translation access] ### 00:17:10 [Dark background with a subtle halftone dot pattern forming a lighter, convex circular area in the center]" ### 00:18:07 # Code Summit AI Engineer Presented by Google DeepMind ## LIVE TRANSLATION Scan the QR code to access real-time translation. [https://events.google.com/codesummit/translation](https://events.google.com/codesummit/translation) [QR code for live translation] ### 00:18:38 [Dark background with a subtle wavy pattern of small grey dots] ### 00:19:02 # Code Summit ## AI Engineer ### 00:19:36 # AI Engineer Code Summit Presented by Google DeepMind ## AVAILABLE # LIVE TRANSLATION Scan the QR code to access real-time translation. [https://events.google.com/io/live-translation](https://events.google.com/io/live-translation) [QR code linking to live translation service]