# Loop Engineering from first principles — Kyle Mistele — session 2026-06-30T22:45:00.000Z → 2026-06-30T23:05:00.000Z

_79 transcript lines · 3 slides · source: full recording_

## Transcript

They're useful. They're scalable. They're reputable, but their operational value is limited. In the middle, their scenario based valuations. These simulate realistic workflows. And at the very top, you see production telemetry. This is where the highest value evaluation signals come from. The surprising insight is that the most evaluation data often comes from real users interacting with real systems. Now let's talk about the offline words. So offline evaluation still matters, but the methodology changes. Instead of evaluating prompts, we evaluate scenarios. For example, a customer support workflow, a cogent workflow, research workflow. The agent operates inside the simulated environment. We measure the task completion rate, tool correctness. And quality. Resource usage, which is which becomes exponentially high at high scale. The key takeaway, agent evaluation should be scenario driven, not prompt driven. Once a system reaches production, every interaction becomes a signal. This is one of the biggest shifts in evaluation thinking. Production traffic is no longer just traffic. It becomes evaluation data. We collect execution traces User outcomes, escalations, failures, feedback signals. Production is the largest and the most representative evaluation data any organization will ever have. For many organizations, we view humans as fallback systems. I think that's the wrong framing. Humans are the evaluators. They provide signals that automated systems cannot. They assess correctness, trust, usefulness, safety. These signals become really critical for calibrating evaluation pipelines and identifying blind spots in automated metrics. The most successful systems combine automated evaluation with targeted Human review. Now agent systems drift constantly. Model changes. You have a new version every couple of weeks or months. The prompts can change. Tools can change. User behavior can change. Hey, everybody. My name is Kyle. I'm co founder of a company called human layer. And I'm here to talk about loops. I think we've all been building loops lately. And I've realized recently, I think we're all doing it wrong. Loops are really powerful. Don't get me wrong. But so much of the discourse around them is hype driven and just really not helpful. Right? We I think we have this idea just kind of as an industry somehow that we can like pipe a prompt in a loop to a coding agent and that we can build software this way. Right? Maybe we're investing a lot of time. And verifiers. Maybe you have six different code review agents. But at the end of the day, if we're doing this, we're still building 40,000 line PR is that just nobody wants to read. Right? And this isn't a throw shade to Jeff Huntley. Right? This is Ralph is an innovative. It's a sharp tool that works very well for certain types of problems. It works very well if you're not building on a team and it works. Very well if you're not working on critical systems, but most of us are working on teams and we don't fit in that box. So today I want to talk about how to build loops that work in large complex code bases for systems that have real customers, real users, real regulatory obligations, and service level agreements, and everything else that keeps us from shipping YOLO 40,000 line PRs straight to production. In other words, I want to talk about how to build loops for the real world. If you're not aware This post actually dates back to July. It went viral this past January, which is when a lot of us, I think started building loops and of course much more recently, I'm sure you are going to see this slide a lot this week. But Peter Steinberger said that we shouldn't be prompting coding agents anymore. Right? We should just be designing loops that prompt our agents. Of course open claws notoriously built on loops loops build the code loops, review the code, they merge and release the code, they find and fix the bugs. There's even loops for finding and fixing bugs in the loops that are merging the things right. It's loops all the way down. It's Boris tourney also the creator of Claude code recently said that this is his entire job as an engineer now is just writing loops to prompt Claude and eventually we might not even need loops. Right? We're just gonna have like swarms of agents designing loops to prompt agents building swarms for loops and like I don't know somewhere we're like writing production code, I assume. And in fact all of our loops that we're building are producing so much code that we can't possibly read all of it. Right? So we might as well just not read any of it. Right? We're investing in verification and in code review. But all this code is read only. This is the thesis of a conference that was here in town last month. So a lot of smart people at the Frontier Labs think that this is the future of software development. And if you're doing this, you're moving 10 X faster and everybody else is getting left behind. Now, it's not clear how well this works yet. To fix the cloud code terminal flicker. The open code team wrote a renderer in a fraction of that time and open claw of course also notoriously has stability issues. What is abundantly clear however is that this shit is really expensive if you don't work at a Frontier lab and have an unlimited token budget and all this code that we're writing is actually really expensive. Right? Matt Pocock talked about this It took six months Recently bad code is much more expensive in the age of agents than it has ever been at any point in the past. So today I want to talk about what I think works in the real world and what we started doing at human layer, which to be clear is still building loops, right? I think loops are super powerful, but we can design loops and still read the code. In fact, we can design loops to make it easier to read the code because the loops are making the code better. We can solve hard problems in complex code bases with loops and we can build our software for three incrementally. Is going to take some real engineering. Y'all. So let's talk about control theory. Control theory is all about how we drive a dynamic system which would be your code base towards some desired stable or optimal end state. Right? But to do You have a sensor that measures the current state of the world. You have your set point, right? The desired state of the world. And the difference between those two things is your measured error. You have a controller that reads that measured. Error and turns it into a control signal about an incremental change to apply to the system. We have an actuator that applies that change to the system which is undergoing disturbances in the meantime and then we remeasure recompute our measured error and we're back where we started. Now this sounds really complicated and it can be. I have a twin brother actually who's an aerospace engineer. This is how they keep fighter jets from falling out of the sky. But it's probably a little bit simpler than most of y'all think. Does anyone have one of these? A thermostat uses a control Loop. Right? For for any of our European friends in the audience, it's part of something we have here in The States. It's called air conditioning. And, most of us probably actually use control loops on a daily basis. Right? Kubernetes auto scaling systems are built on control loops. Infrastructure as code uses a desired state, current state iterative chain. And just like control loop pattern, postgress is auto vacuum and reacts virtual dom both use or approximate control loops. Control loops are ideal when we have a system that we want to change a problem we can measure in a way to get feedback on the result of that change. Like good software engineers have always been taught to do control loops, change the system incrementally instead of just trying to get straight to the end state immediately all at once and risk blowing everything up. Right? They help us to avoid over steering. And destabilizing the system and it minimizes risk. So control loops will be opposite of what I'm going to call a blind Ralph loop there. How we avoid PRS that look like this because nobody wants to review this, right? Which is not to say that all Ralph loops are blind loops. The best Ralphs are actually applying control theory. I know Jeff Huntley is out in the hall somewhere wandering around. If you go talk to him, he's gonna tell you the same thing, right? That Ralph is a teaching device and I think some of us read it a little too Really but this is how we should have always been building loops. But the other issue with Ralph loops, they're not incremental, right? It's just a bash loop. So we have to build a gentle control loops. And to do that we start by defining a set point which is the desired end state of our code base with respect to some property of it. And we had a sensor. There's a lot of ways to build a sensor. It can be strictly deterministic. Your es lint rules, your A. S. T. Grip your pack work or it can be non deterministic. You can Have an agent and a skill and a bunch of natural language rules and you could also just have a pipeline like a combination of the two. So how do we build a genetic whoops. There we go. Now this is all theory right Practically speaking. And because we're using agents, we can blur the lines a little bit between system components. So aid and buys react doctor for example is fantastic. It is it's a great way to catch all of the react slot that Claude snuck into your code base last week. But it's a hybrid sensor and controller. It tells you what are all the problems with your react code. And also by the way, what are the top three things you should fix and how do you fix them? Similarly, our controller and actuator might actually just be a single agent deciding on an incremental change to make and then applying it in the same context window. But I want to zoom in on the controller a little bit because without one or without a well tuned one we might make too large of a change all at once or we might make the wrong change entirely. And if you put That in a loop, you're in trouble pretty quickly. So we can use control loops to root out bad patterns and to clean up our code, but we can actually use them for all sorts of things. Right? We could make sure that our A. P. I. Is compliant with someone else's open A. P. I. Spec. We can make sure that our M. C. P. Server is compliant with whatever version of the M. C. P. Specification that we're currently on. Haven't checked. You could mirror a project from python and just hype script or vice versa. You could even maintain your beat based slop fork of next j s against the upstream. Key questions are can we find something we can measure? Can we apply changes incrementally and can we get feedback on the quality of those changes to illustrate that I'm going to walk through a control loop that we use internally at human layer for our loop. We are incrementally migrating our RPC API to effect. We adopted it for some of our race prone code. We like it. So we're adopting it across the rest of our code base. If you've never seen effects code before the code on the right is just The kind of trivial procedure on the left rewritten in effect. The syntax is really weird. We're psychos. We really like it. It's not for everybody. That's okay. This isn't to talk about effect. So we'll keep moving. Oh, clicker is not working. Cool. So step one, we have to build our sensor to find unmigrated procedures. We can have an agent do this, or we could use grep or rip grip. But instead we're gonna use a st grip because it's really powerful. It's a great tool to have in your toolbox for building loops. It's language agnostic. It's out of band from your typescript config or es lint rules, which if you're a type Developer. You have watched claud disable those with inline comments. But so we can just write a simple rule that finds on migrated procedures based on the pattern above over time. We can even layer on more rules that describe other patterns we want to get rid of with granular including exclude paths. If you have a multilingual mono repo like we do, it will work for any language you could possibly imagine and we can just scan our code base and it will produce a long list of Violations way too long. In fact it will give you about 50 keys per violation. So we're just going to filter it down to four and we're going to sort it deterministically. Why are we doing that at the beginning? I said this was going to be practical and so we're going to step outside of our control loop paradigm for a second because before we start incrementally migrating procedures one at a time, we need to enforce that all new procedures are using effect. Right? So we're gonna run a full scan once on main sort all the violations deterministically and track it in our version control. And then on every new PR we can see if the branch added any on migrated procedures. Right? So this is our control loop and our system is undergoing disturbances in this case. All of our teammates shipping Claude's law and this is how we make sure that they're not undoing our loops work. This doesn't map directly to a part of the control loop, but we can kind of like squint at it a little bit and called a disturbance dampener. So now that we've stopped the bleeding, we can actually design our controller for a simple controller. We could just deterministic quickly pick the first violation from the list You can use bash and j q or we could get a little cleverer and use a s t grep to find the smallest unmigrated procedure and always pick the smallest one to reduce the risk. We could have an agent make the decision if we really want to. I don't think you should ever send an agent to do deterministic codes job, but you certainly can. In fact, depending on the complexity, we could have the agent pick the procedure to migrate and just do it at the same time like we just talked about. But we can make this even more powerful, right? Because we're not just migrating to effect. Sake of it. We're doing it because it's helpful for handling errors and for helping us instrument our code better. And so what we could do if we want to get really clever is we can look at our telemetry and figure out which procedures have the most errors or the least instrumentation or has a gap in our A. P. M. Right? And when we send a control signal to our actuator agent, we can include not just the procedure to migrate, but also all the data about the things that we're trying to fix with this migration so that the actuator agent can actually make the code better instead of just doing a one to one migration. Ma'am, there we go. So next is building our actuator. Our actuator is just an agent plus a skill. Bring your cli coding agent of choice. You should spend a lot of time on the skill. Not all of that should be up front. You'll want to iterate on it over time based on what works at human layer. We like to build out what we call golden patterns by hand before setting the agent loose. These are just like idiomatic handwritten examples for the agent to follow because they're just Pattern replicators and otherwise you're getting what's in the docks or what the agent knows from the internet. And so we pipe the skill plus our control signal into our actuator agent and the skill of course should include a response template and the agent is going to work and work and work and it will produce a final response and then we're going to deterministic lee commit and push and create a PR using the final message as our PR description. Now all we have to do is actually run the loop. Right? My recommendation is to use get hub actions or your git lab or your circle c I or whatever else you're using because it has access to your code. Has access to your secrets and it has great dispatch and scheduling primitives. Right? We don't need a new cluster for this. So we can write a workflow that runs a single iteration of the loop since control, actuate and creates a PR and then we can schedule this to run once a day. And every morning we walk into the office to a small incremental PR that's low risk. And when we first did this, it was actually really frustrating and we turn the loop off and Because we had to constantly update the skill, we had to constantly check out the branch, change the skill, change the code, commit and push and our loop was actually really high friction. Right? But there's a better way to do this where we can put a human on the loop in a really low friction way to re steer it when it goes wrong. And the way to do this is to just create a feedback file that's tracked in version control just as a markdown file. Right? We can deterministic lee load it into our actuator agents context every time that it runs after we run the controller. Then we can add a label to the PR. Right? Each workflow needs to be able to identify PRS that are created since there might be a bunch of different. Loops running and we only want workflows to respond to feedback from comments on their PRS and we're going to add a comment trigger to each loop workflow. So when a user leaves a slash iterate comment on the PR, the loop workflow is going to pick that up. It's going to deterministic lee load all of the PR context, the diff, the comments, the review comments, the description into the agents context along with the skill and it's going to instruct the agent to fix the code. But also to update that feedback file. Right? It looks kind of like this. And the benefit of doing this way is that now that feedback file with instructions is tracked in your version control. You can see how you've changed it over time. You can revert it if you need to. So the next thing we're going to do is that flow control because the other problem that we had when we did this was that if we were at a customer site for a week or if we were traveling or spend six days working on slides instead of writing code, the PRs from all of our loops would just stack up. They duplicate work. They'd conflict. And we wouldn't get around to doing it. Work is important but it's not that important. And so now we just had all this like junk we had to deal with. That wasn't important. So this is actually a really easy problem to fix because each loop and its workflow has a label that gets attached to PRS when the workflow first runs before we check out the code and install the dependencies and run our sense actually or sense control actuate steps, we can just check and see if the last PR that we created And, like, the loops were Or any pr with the loops label on it is open. And if so we just shut down. Right? Because this means that the last time that a human reviewed the code from this loop was before the loop ran, right? No human reviewed the last output. So there's no reason to stack up even more work for humans to review this way. We have exactly one pr at most open per loop at a time. No stacking, no duplication, hopefully no conflicts. And of course once you're feeling confident in the loop, we're gonna want to speed it up. Right? I have 150 RPC Procedures to migrate. If I do one at a time, it's going to take six months, which is way longer than I want to wait. Fortunately, there's a lot of ways to pick up the velocity of our loop. We could have our controller pick three procedures to migrate instead of one at a time or five. We could have our controller pick three or five and then do each of those in a separate implementation phase which will be both cheaper and more reliable since each migration gets its own context window. Or we could just run the workflow four times and give one PR to each of four people on the team. So let's put it all together. We built a control loop that improves our code incrementally and we're actually reading the code. It has adaptive flow control. So we're not creating a bunch of loot or a bunch of work that nobody wants to review and we can resteer it on the fly in a super low friction way. If you want to try this yourself, we built a skill. Please try it out. My twitter handle is down there on the bottom. Please share it. I would love to see what you build. And if you get excited by this at human layer, we're hiring here in San Francisco. And if you're working Mission critical systems and want to figure out how to get more out of Ai we'd love to chat. Thank you so much. In 2026, coding agents will quietly retire their first software platform. Not because it's bad, simply because the platform is unnecessary. I am Dominic Tornow. I am founder and CEO of Resonate. Resonate is a durable execution platform built with minimalism and simplicity as its core technical values. And these properties will play a central role in this talk. At Resonate, we have a working theory where software engineering is headed. General purpose implementations will increasingly

## Slides

### 00:09:53

# Hybrid Sensor + Controller
## Aiden Bai's react-doctor

### Screenshot of `react-doctor` output
The tool reports "ALL 202 issues":
- Bugs:

### 00:10:21

# Engineering the future of AI

The Controller decides what to change - and how much to change at once
The Actuator (Agent) applies the incremental changes to a system

### Loop Engineering from First Principles
Kyle Mistele | @0xblacklight

[Block diagram illustrating a feedback loop with a desired end state, Controller, Actuator (Agent), and Sensor (Rules/Evaluation)]

### 00:10:48

# Applications of Agentic Control Loops
- Eradicate bad patterns
- Adopt new frameworks incrementally
- Maintain a fork against an upstream
- Mirror a project into another language
- Keep integrations up to date
- Ensure compliance with a third-party spec
- Resolve system anomalies (errors, alarms)

Loop Engineering from First Principles | Kyle Mistele | @0xblacklight