# Software Engineering Is Becoming Factory Engineering — Zach Lloyd — session 2026-11-12T17:50:00.000Z → 2026-11-12T18:10:00.000Z _60 transcript lines · 40 slides_ ## Transcript I think that this is gonna change, and it's starting to change already. The the reasons are... There's a few of them. So one is, like, if you are doing local interactive prompting as your main way of working with coding agents, you're kind of limited by human throughput, really. You cannot automate if every agent needs to be interactively prompted. There's a second reason too that I hear a lot from platform teams. From, you know, engineering leaders, this mechanism of, like, everyone choosing their own local setup, their own models, their own skills, it makes it very hard to actually make sure that you're using these agents efficiently, from like a cost perspective and a productivity perspective. And then finally, there's like a, you know, big security problem and governance problem when everyone is setting up and running their own local agents. And so the thing that, is coming, I loosely call a software factory. That means different things to different people. I would... Well, I'm curious for people in this room. Like, how many folks are currently trying to build or building or using their own internal software factories? Show of hands. Okay. So I would say that's almost more than the interactive agents at this point. That's super cool. So I'm gonna define what I think of as a software factory, but I'll start by saying what I think a software factory is not. I think it's not just a loose set of automations. I don't think of a software factory as a collection of AI teammates. I don't think it's a shared set of skills and MCPs, and I also don't think software factories are personal agent swarms. I think these are all useful primitives, but they're not the end state that we should be shooting for. And so to me, the way that I define software factory is very specific. And I'll unpack this, but I define it as a code defined closed loop system for continuously measuring and improving cloud coding agents. That's a mouthful, but let me unpack it. So if you wanna build a software factory and really scale it, I believe the first thing you need to do is fully encode it in a declarative language. So you need to say, okay, these are all of the coding agents that I want to run. These are how they're triggered. And work comes into this factory. This is the workflow from, you know, from triage to spec. Whatever your workflow is, you need to specify what m c p's are available, where your agents run, really as much as you can in a terraform style declarative, you know, set of code files that define your factory. And I'll talk about why this matters in a bit. The second part of it is a continuous closed loop of improvement, and that closed loop can look something like this. This is the typical one. This is just really the SDLC, where you have ideas that come in from people. They could come in from automated signals, and then your agents do some amount of work, taking that issue through various stages from triage to implementation, verification, monitoring, and so on. The important piece here is really this closed loop concept where the more that your agents run, the more efficient you can make this loop. Talk about how you do that in a second. The third really key aspect if you're building a software factory is that everything needs to be measurable. If if you're just going on vibes in terms of how you deploy coding agents, you can't be confident that you're actually, getting the best highest quality outputs the... At the best cost. You can't be confident you're maximizing throughput. So you really need a measurement mindset and measurement capabilities. And so if there's one Thing that I'm hoping people walk away from this talk with. It's the idea that the way to work with coding agents and the way to think about them is that there... There's a really cool fundamental new engineering problem, which is how do you take this magic new primitive that we all have that we've gotten the last year that you have these intelligent agents, and how do you engineer a system around them to use them most effectively? To me, it's not About, like, growing a garden or following a recipe. We're all engineers. We've spent our careers learning how to solve problems from an engineering mindset. And when you think of coding agents as a new engineering primitive, then you naturally want to... You'll naturally start to build a certain way of using them. That's the mindset. So, yeah, just to unpack that a bit, so there's continual measurement and continual learning, but there's also a new Set of tools. And so when I was learning how to become a better engineer at Google, you know, I really had to get good at using things like debuggers and profilers and Chrome Dev Tools. And, basically, there are all of these tools of the trade for building software by hand that maybe are not as important now, but they really separated, the the best engineers from the okay engineers. And in the new world of factory engineering, there is a new Set of tools that everyone should be trying to to master. And so those are things like evals. It's... Can you set up, self improvement loops that work? Can you write benchmarks? Can you AB test your factory? And that's a really important skill set, and that... Mastering that skill set is something that I I think of as, like, factory engineering. Now that doesn't mean that that's the only kind of engineering you're gonna be doing. You're still gonna be doing product engineer. Meaning, you as an engineer should still feel responsible for, you know, does the product you ship actually solve a real user or customer problem? But the best engineers, and I would say, like, every engineer really, is gonna need to learn this new tool set of how do you improve the system, not just build the product. And so what does that look like? If you think from first principles about how do you engineer an agentic system? I think you come up with something like this. Doesn't have to be exactly this, but, it's kind of an infrastructure stack that, that you're gonna wanna use to engineer your agents. And so at the bottom of this is what I call factories as code. I talked about this earlier. It's defining all of your coding agent workflows in a in a code format. And the value of that is that you get the ability to branch off it and test different factory configurations. You can roll back. And then crucially It it opens up the way for self improvement, which is the ability for coding agents themselves to suggest changes to how your factory works to get that continual loop going. Above that is a layer of data and context. This is also crucial. So if you are running coding agents at scale in an organization, you really must be recording every... Everything that those agents are producing. So all of their conversations, all their artifacts, the video Is that they make the screenshots. And the reason for doing that is that's the raw material that you will use to improve the way your factory works over time. On top of that is a compute layer. So you need to have sandboxing. Sandboxing increases security. It makes your factory more repeatable. Your sandboxing should be probably multi platform. On top of that is the inference layer. The way that I would think about inference and models in a factory is they For a degree of freedom to test. And so your factory, however you build it, should support multiple models. It should also support multiple coding agents because, you know, cloud code might be better for some things. Codecs, py, warp, these harnesses all have different properties. You might want to write your own coding agent harness that has some of your organizational context baked in. So your platform should support that. On top of that, you need an orchestration layer. So this is what actually Moves work through the factory. In some ways, the least interesting part, but you need that. And then finally, your factory should be all API and MCP first so that it can be used not just by, you know, people, but it can be used by other coding agents, and it can go to where the work is being done. So your factory should integrate into things like Slack, linear, Jira, GitHub, and so on. I'm gonna give a quick demo of what some of these tools look like in the context Of warp factories. But this stuff I'm going to show, I think, applies also if you're building your own. So let me start here. So at the foundation I mentioned earlier is factories in code. And so I'm gonna show some code. It's not gonna be that interesting. But if you're familiar with something like Terraform, you will get what's going on here. You basically want a declarative way of saying, this is my Factory. My factory, has certain policies. It accesses certain repos. It has certain triggers. It has a sort of blueprint that defines the workflow of work going through it. And then if you zoom out, all of these other things that are part of the factory also have code definitions. The second thing, moving up a layer, is the idea that, sorry, that everything that you do In the factory, should be tracked across your entire team. And so think of what I'm showing here is this is a Google Docs style live view of what every coding agent on Warp's team across all of our engineers are doing at all times. And so I could click it, and I could see, Okay, I don't know what Maggie is working on, but I know I could look at it. She's implementing Opus 5.5 on the high plan. She's doing it from Slack. It has this artifact. I could look in and see the actual conversation here. I could see what it's generated. And so I could see this in real time, which is cool. But the really important thing is that you gather all this information so that it can be used for improving the factory. And so that brings me to the third thing moving up the stack here is once you have all of this information that's being stored centrally, you get the ability very easily, without doing any Install stuff on everybody's laptop, anything like that, to understand how these coding agents are working. And you can think of that in terms of traditional output metrics, like your DORA metrics, like, Okay, cycle time. But you can also think of that in terms of how automated am I, like how many human interactions per PR, what's your throughput. You get, you should be able to see data very clearly here on, like, what's your cost. Again, surfacing this, If you don't have this data, you're kind of flying you're kind of flying blind. So this is really important. And then there's this next layer of data, which is also really important that I mentioned earlier, which is the ability to set up online evals that run across all your coding agent sessions. And so what I mean by this, an eval is just like as these coding agent sessions are running and they're recording their output, you want, the ability to to go Go back and have another agent look at them and score them on some dimension that might indicate how well they're doing. And to make that more concrete, so it's like we have an eval, a score here that we use that is looking for redundant tests. This is a good example. And so one common failure mode of coding agents is that sometimes they will write tests that are unnecessary. We want to catch that. And that's what this eval does. We want the ability then to see, okay, when it's Writing too many tests, why is it doing that? And is there a mechanism that we could change in the factory for improving that? And so, again, just to kind of recap here, you want a way of recording all the runs. When these runs run I didn't show this. But all of the runs are you want a way of specifying what is the actual loop that they're running through. You want to be able to see where they are in that loop. As the runs, work, you want to be able to gather their data. Once you gather their data, you wanna be able to set up these online evals. And then these online evals give you a sense of, like, okay, where are there opportunities for improvement? Once you have that in place, you can do something that we call self improvement, which is rather than having humans look at the outputs of these scores, you have agents look at the output. And so the way to think about self improvement is it looks at the score output. It says Okay. We saw some number of failed runs. Let's try to look for patterns in those failures and suggest a possible solution. And then the form that that solution takes is as a PR against your factory definition. And so this is like the awesome thing about having your factory defined as code and being able to use coding agents is that you can then have the coding Diagnose problems with how they're working internally and suggest fixes to them. The final tool that I'm going to point out that I mentioned earlier is the ability to create and run benchmarks. This is also essential. So if you're following this course online, you'll often see things like, okay, this is the best model. This is the new best skill. This is the best harness. The only way to answer those questions for real is to test on your own data. And so you should have in place some Mechanism where you can basically create a benchmark set out of your real data and then replay those tasks oops, I'm on the wrong one here and replay those tasks against different configurations to see how they would have done if you had varied some model or varied some harness or varied your factory configuration. So just to make this clear, This is something that looks like what you might get, when a a new model drops, like in sweep bench or terminal bench. It's showing the trade offs of different models here in terms of cost and quality. You want to be in this upper left hand quadrant. But it's on your own actual data. And you can you can test not just models, but also you want to be able to test, hey, if I got rid of these skills, does my factory perform better or worse? So these are some of the tools. Just to return to this big concept, like, this has worked well for us. We have managed to use these tools to drastically improve our throughput and lower our cost. We we are just constantly performance baseline, we're scoring, we're benchmarking, we're testing, and we're doing factory engineering. And so that brings me back to this last point that I just wanna emphasize again. Engineer your agents. It's a new engineering primitive. We're all engineers. Think about how you can get the most out of them and build the system for that. I think this is very exciting. I know there's a lot of, like, angst around what it means to be an engineer right right now. I know we're not writing code, but I'm still using all the engineering skills that I spent my whole career learning. And so to me, it's still a really exciting time. If you're interested in more factories, you can check it out here. Thank you all for coming to the talk. What happens when AI agents can build entire creative applications in just weeks? Our next speaker is the founder of ArtCraft. A project exploring what's possible with AI powered software development. Ladies and gentlemen, please welcome Brandon Thomas, CEO of ArtCraft. Hey, everybody. Thanks. For attending my talk. I had to put this together late last night. I've been in a whirlwind after this project went viral of unblocking thousands of open source developers. We've had VCs approaching us. I've been in constant meetings. Like, this has been a sleepless month of activity since this happened. So if you're not familiar, I kind of cloned Adobe Photoshop and a bunch of other apps, and that went super viral. My phone, my inbox, everything has just been Absolutely lit up. And so I I I apologize for the structure of the the the the talk. This was thrown together as quickly as I can, but I've got a lot of really exciting stuff to talk about. And I think it talks about the future of not only the industry, but just technology as a whole. So, we're gonna open source the entire world. Like, why not? It's it's something that I've always wanted to do as a developer. That's something that, a lot of us in the open source community are interested in. Open source Powers, a lot of the, you know, technology that we're familiar with, but it's never been at the forefront. Like, Linux has always been like a second class citizen. We've been pulled into a whole bunch of platforms where we don't really have ownership, and AI kind of gives us a chance to take the web back, if you will. So, I might have to do this manually. Cool. So, a little bit about myself. I've been an engineer for about fifteen years. I spent eight years at Square doing, heavy hitting infra. So online payments, the, the authentication stack. Basically, anything that could take down the company, my team was kind of in charge of, and it was, you know, eight years of carrying the pager, a lot of really heavy hitting infrastructure work. And then I've spent seven years doing start ups, and that's a whole different world. It's a lot of risk, a lot of learning, like, what the market needs, a lot of, correcting your own assumptions. ## Slides ### 00:00:07 # Interactive prompting ## Software Engineering Is Becoming Factory Engineering Zach Lloyd / Founder & CEO Warp [https://warp.dev/](https://warp.dev/) [Photograph of a desk with multiple computer monitors displaying code] ### 00:00:36 # Software factories are coming. 1. Automation increases throughput 2. Centralization for measuring and improving quality and cost 3. Security and governance in the cloud ## Software Engineering Is Becoming Factory Engineering Zach Lloyd / Founder & CEO Warp [https://warp.dev/blog/software-factories](https://warp.dev/blog/software-factories) ### 00:01:07 # Software factories are coming. 1. Automation increases throughput 2. Centralization for measuring and improving quality and cost 3. Security and governance in the cloud ## Software Engineering Is Becoming Factory Engineering Zach Lloyd / Founder & CEO Warp [QR code for live translation] ### 00:01:40 # Software factories are coming. 1. Automation increases throughput 2. Centralization for measuring and improving quality and cost 3. Security and governance in the cloud ## Software Engineering Is Becoming Factory Engineering Zach Lloyd / Founder & CEO warp [QR code for live translation] ### 00:02:06 # What a software factory is not: - A loose set of automations - A collection of AI teammates - A shared skills and mcp setup - A personal agent swarm ### 00:02:36 # What is a software factory? A code-defined, closed-loop system for continuously measuring and improving cloud coding agents. ## Software Engineering Is Becoming Factory Engineering Zach Lloyd / Founder & CEO Warp [QR code linking to https://www.warp.dev] ### 00:03:09 # Factories as code: The way a factory engineer defines and improves a factory. ```yaml schemaVersion: v1alpha1 name: web-server-factory description: ### 00:03:37 # Software Engineering Is Becoming Factory Engineering ## A cloud software factory - New product idea or bug reported - Triage - Spec - Reviewing the spec - Implement - Reviewing ### 00:04:09 # Measurement > Vibes ## Benchmark Results - **gpt-5.6-sol (high)**: Average cost: $6.13, Correctness: 95% (Benchmark ### 00:04:37 ## Software Engineering Is Becoming Factory Engineering - Prompt your agents. - Manage your agents. - **Engineer your agents.** Zach Lloyd / Founder & CEO warp [https://warp.dev/](https://warp.dev/) [QR code linking to warp.dev] ### 00:05:09 # Software Engineering Is Becoming Factory Engineering - Prompt your agents. - Manage your agents. - **Engineer your agents.** [https://warp.dev/](https://warp.dev/) ### 00:05:41 # The factory engineer toolbelt: - Blueprints - Automations - Evals - Self-improvement - Benchmarks - A/B tests ## Software Engineering Is Becoming Factory Engineering **Zach Lloyd** / Founder & CEO Warp [QR code for live translation] ### 00:06:06 # The factory engineer toolbelt: - Blueprints - Automations - Evals - Self-improvement - Benchmarks - A/B tests ## Software Engineering Is Becoming Factory ### 00:06:38 # Product engineering and factory engineering. **Product engineering** = Building the product **Factory engineering** = Improving the factory that builds the product ### Software Engineering Is Becoming Factory Engineering Zach Lloyd / ### 00:07:09 # THE FACTORIES STACK ## 06 ACCESS - VIA Factory MCP: Any coding agent - Comms Platform: Slack + Teams - Task Tracker: Jira + Linear - Code Forge: GitHub + GitLab - Any Device: Web / mobile - Programmatic: API / SDK ## 05 ORCHESTRATION - Native Orchestration: Triage / Spec / Impl / Review / Verify - Multi-agent: Warp + Claude + Codex - Human in the loop: Live steering + handoff - Automations: trigger / cron ## 04 IMPROVEMENT - Metrics: Autonomy + cost + cycle time - Scorers: Evaluate completed runs - Benchmarks: Compare models / skills per task set - Self-improvement: Automatic observer loop ## 03 INFERENCE - Warp-hosted inference: Multi-model + frontier + open-weight - Bring your own inference: Bedrock + Vertex - Custom model routing: LLM routing rules + self-improving ## 02 COMPUTE - Sandbox: Docker ### 00:07:37 # THE FACTORIES STACK ## 06 ACCESS - VIA Factory MCP: Any coding agent - Comms Platform: Slack, Teams - Task Tracker: Jira, Linear - ### 00:08:09 # THE FACTORIES STACK Warp hosted or customer hosted ### 06 ...... ACCESS - VIA Factory MCP - Any coding agent - Comms Platform - Slack · ### 00:08:37 # THE FACTORIES STACK ## 06 ACCESS - **VIA Factory MCP**: Any coding agent - **Comms Platform**: Slack, Teams - **Task Tracker**: Jira, Linear ### 00:09:06 # THE FACTORIES STACK - **00 FACTORIES-AS-CODE** - `Factory.Yaml` - Agent Definitions - Automation Definitions ### 00:09:37 # Software Engineering Is Becoming Factory Engineering ## Dashboard [Screenshot of a web application dashboard for `eng-factory-slim` showing productivity metrics. It includes a line graph of human interactions per ### 00:10:06 # Software Engineering Is Becoming Factory Engineering ```yaml blueprint: foreman start: to: - stage: research when: The request is a bounded informational question ("can ### 00:10:39 # Software Engineering Is Becoming Factory Engineering ## Zach Lloyd / Founder & CEO warp ### Application: Runs [Screenshot of a web application interface displaying a list of "Runs" or tasks with a navigation sidebar] [QR code for live translation] ### 00:11:10 # Software Engineering Is Becoming Factory Engineering ## Zach Lloyd / Founder & CEO warp ### Runs eng-factory-slim Implement Opus 5.5 High Plan [https://live.warp.dev ### 00:11:38 # Software Engineering Is Becoming Factory Engineering ### Dashboard: eng-factory-slim - **Productivity** - Human interactions per PR: **19.4** - ### 00:12:10 # Software Engineering Is Becoming Factory Engineering ### Application Screenshot: eng-factory-slim Dashboard - **Zach Lloyd** / Founder & CEO **warp** [https://warp.dev/]( ### 00:12:42 # Software Engineering Is Becoming Factory Engineering ### Screenshot of eng-factory-slim Dashboard Zach Lloyd / Founder & CEO warp [Screenshot of a web application dashboard displaying productivity, spending, and scorer metrics ### 00:13:07 # Software Engineering Is Becoming Factory Engineering ## Screenshot of Warp platform: Redundant Tests [Screenshot of a web application showing a table of "Redundant Tests" results, with columns for ### 00:13:38 # Software Engineering Is Becoming Factory Engineering ## Zach Lloyd / Founder & CEO warp [https://warp.dev/](https://warp.dev/) [Screenshot of a web application dashboard titled " ### 00:14:06 # Software Engineering Is Becoming Factory Engineering Zach Lloyd / Founder & CEO **warp** [Screenshot of a web application dashboard titled "Self-improvement" showing a list of tasks or issues with descriptions ### 00:14:38 # Software Engineering Is Becoming Factory Engineering ## Zach Lloyd / Founder & CEO warp ### Screenshot of a web application showing "Benchmarks" [https://warp.dev/](https://warp. ### 00:15:10 # Software Engineering Is Becoming Factory Engineering ## WarpBench 0.1 Synthesized, self-contained implementation-agent tasks distilled from real Wilson factory runs. Ten easy, nine medium, five hard ### 00:15:35 ### Benchmarks: WarpBench 0.1 Some plotted scores are based on incomplete scorer results. Use Rescore missing scores in the three-dot menu next to Run to retry unsc ### 00:16:06 # Factory proof points: - **40%** **MORE PRS MERGED** Merged PR volume increased from 1,682 to 2,3 ### 00:16:39 # Software Engineering Is Becoming Factory Engineering ## How I feel about being an engineer today. Zach Lloyd / Founder & CEO warp [QR code for more information] ### 00:17:06 # AI Engineer ## Code Summit - **Sponsors**: Unblocked, AWS, Google DeepMind, Devin, databricks, Render, SENTRY, CISCO, OpenAI, AN ### 00:17:38 # AI Engineer Code Summit Presented by Google DeepMind [A grid of sponsor logos, including AWS, Google DeepMind, Devin, Databricks, Render, Sentry, Cisco, Anthropic, OpenAI, Pinecone, Unblocked, Sonar, Factory, Atlassian, Exa, Overout, Vercel, Firecrawl, Paradot, Datadog, Judgment Labs, Key, Cloudflare, and M.WireThread.] ### 00:18:16 # AI Engineer ## Code Summit ### 00:18:37 # the wo - Brandon Thomas - AI Engineer [Faint wireframe geometric shape] ### 00:19:09 ### A TALK BY BRANDON THOMAS # How to Open Source **the World** - Brandon Thomas - AI Engineer Code Summit - November 12, 2026 [Geometric wireframe shapes (polyhedra) as background elements] ### 00:19:38 # How to Open Source **the World** ### A Talk by Brandon Thomas - Brandon Thomas - AI Engineer Code Summit - November 12, 2026 [Abstract