# We're the bottleneck, but we don't have to be — Ido Salomon — session 2026-06-30T21:25:00.000Z → 2026-06-30T21:45:00.000Z

_61 transcript lines · 16 slides · source: full recording_

## Transcript

There's also a two UI. So lots of fun. Stuff, with a to a protocol, but, this is kind of what's worked for us. So sharing that with with you folks. Feedback and evals. So here the quote is, shipping is the start, not the finish. So what we do here, on the agents team is we have kinda multiple ways we do evals and collect feedback. Obviously, you know, we'll have folks, call in or or email us or or just let us know and tell us. But the main way is we have this thumbs up and thumbs down mechanism. And here, Someone is able to tell us, hey. This this worked really well. This was a great response, or that wasn't a great response. And that signal we take and we're able to iterate on, and we could take it back and help improve, you know, the response in the future. We also have automated evals. So in in the in our CI, we we have evals that run against real completion so we could test the prompt against, hey. Did it hit Some tools, did it do what it's supposed to do, and that also helps with our accuracy. So, those automated evals in conjunction with collecting feedback really help us, improve our our our tools, our skills, our harness, and and that's really how how we're able to iterate so fast and so quickly. Humans in the loop. So this is a really cool feature we built where we deterministically interrupt the agent loop if there is a tool call approval required. So Okay. Hi, everyone. I'm Ido Salomon. I am the creator of AtriumCraft, and I also created MCP UI, which became MCP Apps and MCP The steering committee. I've done a bunch of other stuff, but we're here to talk about why we're the bottleneck and why we don't have to be. So, as you all know and use every day, agents are amazing. There's so much they can do, from generating cat memes to b2b SaaS apps. So if they are so amazing, we have to ask, why aren't we all unstoppable? Why doesn't everyone have an army of agents that just does whatever they want? So it should be relatively easy. I mean, you just have a bunch of monitors. Then you spin up 25 cloud code agents, and you're done. So the reality is that it's just not that simple. To do something like that, we need to, a, really want to use agents. And secondly, once you actually spin them up, each one of these agents requires you to steer them, direct them, review them, and when it's at scale especially, it's just exhausting. We need to find the skills to allow us to do it without burning out. So in reality, we are the bottleneck. And the the cool thing here is that while we are the bottleneck, the skills that we need to do these things, to find ourselves not burning out every time we use an agent while we are still along, if you try to compare it to other stuff that you've been doing outside of programming, for example Playing video games, it's not really that different. Like, if you're playing Warcraft, for example, or Sims, and you have a bunch of different, units running around, and you need to understand what you're doing and supervise all of it, how different is it from from actually inspecting agents? So I took this concept too far, and I created AgentCraft, which is a game inspired orchestrator that tries to bring all that good stuff together into work. So let's take a quick look of what that actually looks like and try and understand the journey that we can go through to really raise the ceiling of how humans and agents can collaborate. So, this is AgentCraft. There's really a lot to unpack here, so we'll just go through the basics really quick. What you see here is an agent. It's an actual representation of Cloud Code or Open Cloud, whatever agent that you can imagine. It can be detected on your device, or you can actually spawn new ones directly from agent craft. So, like I said, you have, like, codecs and, open code, etcetera. At your command. And these agents can do everything. They're just normal agents. So I can just prompt them in this nice side panel that also has voice, so there's multimodality here, and just do something for me. Prompt. Prompt. And now these agents can do whatever. And if you look around You would see that the map has a bunch of different elements. So we have buildings. These buildings represent functionalities. So, for example, if you can manage your plugins or skills, you have, like, integrated terminal, integrated git. You can do all of your work from within this space. So now that we have this and we understand the basics, let's look at what we can do to raise the ceiling. So the first step It's obviously visibility. I need to understand what's going on. What are my agents doing? So I have this nice side panel, and it shows me the different agents and what their task is and what the last thing they did is and what they're currently doing. Everything I need to quickly understand who needs my attention and who doesn't. The second part of it is we're not just limited to this. We have a map. So I can take, let's say, my file system, project it onto the map, and now I can see exactly what my agents are doing, not just by looking at a list of stuff. I can actually see, okay, this agent is now working. On this directory or this file. And these files are represented as runes on the map, so I can really look at each one and see exactly what each agent did and when. And now I can take business information, because we are the orchestrator, and use that to create lineages. And now I can create cool stuff like heat maps, so I can really easily see what's going on. And The the and the other aspect is that now that we have this visibility, that's not really enough. We need to be able to react to what's going on. If someone needs my attention, I need to be able to quickly jump to it. So much like video games or RTS games or Civilization, whatever, you would click, space bar, and it would just take you to the nearest thing that actually needed your attention. So you can really easily understand what's going on and jump from one thing to the other and answer questions or approve plans, etcetera. I mean, it's a nice step just to have that visibility, but it's not really enough. It's pretty exhausting to keep looking at what the agents are doing in the granularity of what files are they using and what tools are they using at the moment. So we need something else. And the reason is that we have, like, two two main problems here. One is that there's just too much to hold in our heads, even if I can understand what's going That was nice. On, and I can see everything. I can't really do, like, 20 things at the same time. I can't even think of the next thing I want to do if I have this, you know, huge coding project or whatever. So I need to find some way to fix that, so we can. I mean, we can tell the agents, you know, go through my codebase, find out what the next thing I need to do are, create these tasks for me. I just click a button, I accept the quest, and there's some agent that does it for me, which is fun. It's cool. But the next problem here is that, okay, now I have 20 stuff going at the same time, and I have to babysit. Each and every one, and that is completely exhausting. So you could ask, okay, we have agents. Agents are amazing, as we've established. So I can just go ahead and tell the agent to do it for me. Now it's called software factories, but at the time, it was just doing stuff autonomously. So you can give it, like, a general sense of what you want. It would spawn an orchestrator and break it down for you and run It's all in well, in this case, it's a local, container. So everything is just done in isolation, and I don't need to do any babysitting. The agent does it for me. On top of that, you have loops, which everyone really likes right now. So you can do cool things like just tell that to, I don't know, scan Twitter for cool stuff you want to build or scan your GitHub, and it will just create these nice, features for you in the background without you needing to do too much up upfront. The And thing is that I mean, this is nice, but what do I do when I have, again, I don't know, I would say 20, but even five things to review at the same time? It's very, very hard to get that out of the way. So what I did is create, this, like, review kit. So I get the entire changes I need, and I get, what files change, if you want to go, like, file by file, or you can get just get visual evidence. So you can actually see videos of what changed or photos of what changed, so you can really quickly understand what you want or even run a few. Instances in parallel and just pick the best implementation out of these. So now that we have this, we have visibility, and we have autonomy. It's pretty cool. Like, we reached a pretty good place in raising the ceiling of what we can do with agents, but it's still pretty exhausting. I mean, it it was fun, but now I have to ask myself, do I really need to be the Anyone doing this. It's pretty lonely to just talk to agents all day and trying to and, you know, if you look at, what a team does, for example, if it's just me, that's fine. But if I have a team, I might not even be the right person to review what's going on. So I need something else, and that something is collaboration. We can just have other people join us. So the way that looks in agent craft is that I can create these, you can call it, you know, war halls, war room, whatever. Host them. It's like Local, but you can do, like, tunnels, and people can join you. And so I can create a room, and let's say I have a product designer in my team. In this case, it's my wife, and I can approve, her. Otherwise, she'll be mad. So I can approve her, and she does her thing. She designs whatever. And I, as an engineer, can not only see that that's that's her. I can not only see that on IMAP, I can actually take that and follow-up off of it. Prompt. Prompt. Yeah. That just like her. So I can take that design, follow-up on it, implement it while she keeps working. And the cool thing here is that not only can I hand off or fork and kind of have this shared workspace, so I'm not really bound anymore to Git? I can use Git, but I'm not really bound to something. Everyone is working together. You can see that there's also some kind of chat between not only between, the people within the room, but also Between agents. So we can have this really nice notice board of exactly what each person is doing, what each agent is doing. They're all aware of each other, and they can collaborate much more closely, than you normally would be able to do. So for example, here I can say, okay, I took off this design task. So now we have oh, by the way, this is also from devices. You can have, like, mobile and If you like Telegram, you can use Telegram, whatever floats your boat. So we have visibility, and we have autonomy, and we have collaboration. So we managed to raise the ceiling. It's now possible, especially if you're power users, to do really cool stuff and use more agents and do more things. But the interesting thing here was, you know, if I looked at the feedback I got, it's not just that power users were able to do more. The cooler thing for me was that people that had even nothing to do with agents or nothing to do with productivity, come and say, okay, my kids Really loved it, and they use it all the time to to orchestrate agents, which they didn't do until now, or, you know, someone that flunked out of college because he played StarCraft, and now he can suddenly do really productive stuff with it. And a bunch of other examples of kind of bringing communities that so far were kind of behind into this new world of productivity. So we need to raise the ceiling, but We also need to, lower the floor. We need to have some kind of way to take, you know, probably around ninety percent of the people in the world and bring them into this hygienic future without having them burn out. So, again, I took it too far, and there's this, currently named TBD, potentially loopers. The idea here is, okay, why don't why do we need something like, you know, Warcraft, which is potentially Very complicated, especially or might be even daunting if you're not into the granularity of what files are there or how, complex the UI is and kind of try to find a lower denominator of what makes stuff accessible, especially now that agents are much more capable. So instead of granularity or having, like, exactly what files they're working on, maybe I can have it more of a mobile game level, have projects that I keep returning to, and use Oops and other stuff that rely on agents being more autonomous to have, like, a very basic level of, I prompt, I get the result, I don't really care, perhaps, about the way I can see it if I want to, and have this really nice interactive and cool experience that makes me want to come back to this. And, obviously, I mean, this is one visualization. It can really resonate with some people but might not might not present to others because we're not all the same. And what may appear Like, amazing and easy to one person can appear very difficult to another. But here, I'm trying to look at a way to make this more customizable, try and understand what it is that you as a person need in order to understand and want to use these tools. So this is a very, very early draft, but if you want to use it, please let me know. So we're the bottleneck, and I think we've established that pretty well until now. But the skills that we need to stop being the bottleneck and enable this new future are already with us. So we are the bottleneck, but we don't have to be. And I hope that you can take what's relevant to you in your life and apply it forward. So I would ask you for two things. One, AgentCraft is already out there. Like, you can just npx install AgentCraft as a website, and do your thing. This new thing that I just showed you, potentially loopers. Is still experimental. I'm still trying to understand the interest and what people are actually looking for to, get the floor that they want or the ceiling that they want. So please, go here and express your interest and tell me what you're looking for so I can know what to do with this. And thank you. CEO and cofounder of Starlight Search, and today my talk User signals die at retrieval boundaries. So look into, what are agents essentially why agent fails, what is, the cause of fields in retrieval particularly, and how to make actually signals cross the retrieval boundary and how to make your agent basically up compare. So, let's get started. What is an agent? An agent is an element that has agency to reason, involve tools, interact with the real world, retrieve, the memory to complete the task. One major loop here is missing is, learning. It should also learn from what worked and what didn't work. Suppose, if I have to, explain what is agent, like, and explain with react agent. So if I have to explain, what agent does, I have to explain it with React agent. So basically, use it from the agent. Execute, get in a loop, client to a retriever, search, and then pause when the task is complete. This is very basic React architecture. One thing that is missing is how to make agent learn from the outcome. So agent keeps failing at the same task. Got an 85% of AI just failed. So it's and we can 2025 report. The problem came out to be, most of the time, is that retrieval is static. 73% of, our app pipeline fails because of retrieval node generation and context stuffing. So a recent, post from Ram Shrihaisha, the ex CEO of Pinecone said, we have been optimizing for the wrong thing. You are paying a lot for your agent's memory. This is probably broken, and we have been optimizing for the wrong things. We made wrong answers. Does appear faster and cheaper that we forgot to make retrieval learn. So why this is does does does it matter? Again, the third problem is agents are not out of my phone. So there's a missing layer between evals and action. Your observability has all the traces, all the stat that capture observability. The stand that capture every tool call, every error in completion, every exceptions. Your EVA suite judges whether the final output was correct or wrong, basically pass or fail. But these evils are not reflected in agent context, skills, MD files, or agent action in any ways. Right? So the agent doesn't have any access to Why yesterday's runs, runs past or fail? EVA signal dies in the dashboard. This is a missing layer, a system that consume traces, absorb EVA, and convert both into retriever guidance for future runs. So there's a manual improvement task. An engineer actually has to sit and see if the eval and also ability perform well. Rewrite the prompt, redeploy it, upgrade either upgrade to expensive model, restructure to or harness or fine tune the custom models. I our current memory is failing. Why memories was designed to actually address this, but, it's not. So let's see, what we have in as a current system, and current memory is that they basically store user preferences, profile, conversational history, or long lived birth.

## Slides

### 00:02:49

# Agents are **magic**
[Illustration of a blue furry character (Super Grover) flying over a moonscape with Earth in the background]

### 00:03:10

> Saksham (@sxmawl) - Jan 16
> we just hired our 25 founding engineers

## Engineering the future of AI

[Screenshot of a tweet showing a person coding on two monitors]

### 00:03:40

# These skills were **with us** all along
# Engineering **the future** of AI

### 00:04:19

# AGENTCRAFT
## Putting the Orc in Agent Orchestration
[Illustration of a fantasy-style control room where orcs operate computers monitoring a glowing world map, with a central orc in

### 00:04:46

# Engineering the future of AI

## MY AGENTS
- All 4
- Awaiting
- Working 3
- Idle 1
- DISMISS ALL
    - Mag

### 00:05:23

AI Engineer
World's Fair
PRESENTED BY
Microsoft

# Engineering the future of AI

messages yet
Message...
DEFAULT
PLAN MODE

[Screenshot of a fantasy real-time

### 00:05:47

# Engineering the future of AI
[Screenshot of a real-time strategy game-like interface, showing a map with buildings and units, alongside panels for "MY AGENTS" listing tasks and

### 00:06:18

# Engineering the future of AI

## AI Engineer World's Fair
Presented by Microsoft

### My Agents Panel
- **Tabs:** All (5), Awaiting (1), Working

### 00:06:43

# Engineering the future of AI

## MY AGENTS

-   **ALL** (5)
-   **Awaiting** (1)
-   **Working** (3)

### 00:07:20

# AI Engineer World's Fair
#### PRESENTED BY Microsoft

### Engineering the future of AI

### AgentCraft Application Interface

#### My Agents
- All 5 (Awaiting 2

### 00:07:39

# Engineering the future of AI
Visibility

[Fantasy-style map with mountains, forests, rivers, and several castles. One castle is highlighted with a glow, indicating "Visibility".]

### 00:08:18

# Engineering the future of AI

## AI Engineer World's Fair

[Screenshot of a complex application interface with sections for "MY AGENTS", "QUESTS", "CAMPAIGN", "TR

### 00:08:44

# Engineering the future of AI
[Screenshot of a complex application interface with panels for "MY AGENTS", "QUESTS", "CAMPAIGN", "TRACES", "ANALYT

### 00:09:14

# Engineering the future of AI

## Application Interface

### My Agents Panel
- ALL 7 | Awaiting 2 | Working 3 | Idle
- DISMISS ALL
- DECOMP

### 00:19:12

# The missing layer between evals and action

## Observability
Your observability stack captures every tool call, every LLM completion, and every exception.

## Evals
Your eval suite judges whether the final output was correct.

The GAP

## Agent
Context, Skills and .md file

[Diagram showing three blocks: Observability, Evals, and Agent, with a gap labeled "The GAP" between Evals and Agent.]

### 00:19:44

# The Manual Improvement TAX

- Rewrite Prompts and redeploy
- Upgrade to expensive models
- Restructure tool-call harness
- Fine-tune custom models