# Perception Agents — Antje Barth — session 2026-07-01T16:45:00.000Z → 2026-07-01T17:05:00.000Z _62 transcript lines · 39 slides · source: full recording_ ## Transcript Good morning. It's so great to be back here at the AI Engineer World's Fair. Just a year ago, the hard problem was getting an agent to find a button and click it on A screen, especially screens it had never seen before. Now, agents can drive browsers, and they're starting to also drive desktop apps. But what we figured out, clicker clicking was actually the easy part. What we didn't solve is the actual work. And what do I mean with this? Let's take a very simple example. A new team member starts on Monday, and maybe your job is to set up their accounts, add them to a Slack channel, book intros with colleagues, order the laptops, etcetera. And nobody really owns this end to end process in the company, and it might be also touching five different systems. Now Agents can most likely perform each single individual individual step of this workflow. But agents still struggle to do this end to end because the real work lives within the seams of all of those different applications, of all of those different steps you have to take. And this is mostly where it all falls apart. The agent can use every single tool you give it, but it's Still can do the full work. So why do we see this gap? Think about for a minute what we actually built. We taught computers to use computers. So what do I mean with this? We started building out the basics. We taught them clicking, scrolling, typing, calling an API, filling out a form, and we got those These steps really reliable, and you can string them together in a workflow. And agents these days are fairly good at, like, operating those workflows. So why can't you not just hand them more of your work and then literally just walk away and trust it to be completed? So all the things I talked about, like using a tool, models itself, tool use Stringing agents together, this is all capabilities. And we mostly figured out how to add capabilities to models. Now the next hard part is really reliability. And without reliability, we cannot really build up trust in those systems. So here's a quick gut check, and maybe all of you can just think about an agent doing work in an End to end workflow. How often do you think that actually succeeds these days? Maybe 60, maybe 80% of the time? And it sounds really fine. But if you look into this, if your agent, one in four times, deletes the database, you will never touch that agent again. Right? So when you Need this reliability, you really need to be it in the nines. You need to have the trust that it actually can do the work successfully. Now, there's actually one place where we made enormous progress on reliability and trust, And this is coding. Right? Think about how fast coding evolved. I still remember the first time when it started auto completing for you. Right? You just tapped auto complete. Amazing. Then, short time later, it started to write functions, and we thought that is amazing. And now look at these days, coding agents write the code, they open up the pull request themselves, and we had it earlier this week, code keeps flying by. So once in a time, we were able to Every single line that it generated, we felt, like, the urge we need to really read it and make sure it's correct. Right? I think most in the audience here can still relate to that. These days, I think hardly anyone is still doing that. Like, we cannot even do that. Right? Code is generated at such a pace. At the same time, coding made the jump. So why is that? Because we were able to bring it from just being Capable, the coding agents, to actually be reliable and then trusted. So why is that? Why was coding first solved? It's because code is verifiable. You can run it, you can test it, you can check it, and you can be for sure that it worked. So reliability showed up in the first place. You can actually Verify the answer. But here's the catch. Most of the work we do, if you look at the broader knowledge work areas, is not like that. KnowledgeWork is messy, and heck, the whole real world is really messy. Did the report I created land? Is the design on brand? Did it get it what I Actually meant. So there is no unit test that can answer those questions. So verification really hits the wall right where most of our work lives. It's living in the seams of all of those applications we're using on a day by day basis, and nobody really has cracked this part yet. How do you make an agent reliable when there's no way to verify the answer? That easily. And that's a field that is still wide open. So how can we solve this? Well, so how do humans handle messy work? I mean, we're successful at it. Right? Each of us, like, every day, we work across different systems. We manage out how to onboard a new colleague. We do this. Well, we're doing it by figuring things out together. You grab a colleague, you jump on a Zoom meeting, you're discussing things, you're looking at the problem to solve, you're discussing p pointing at systems, and maybe two minutes later, you solved it. You're done. But none of this work is actually directly verifiable, and we do this all day. So, one of the things is we're looking mostly at the same screen, right? If you're jumping on a meeting with a colleague You see the same screen, both of you, and you can actually, like, figure out really quickly what needs to be done. So this is what the agent these days is missing. You don't necessarily need a bigger brain. What you need is this shared context because if we're looking the agent and myself at the same screen, I probably have much less explaining to do. So what kind of agent do we really need to build to achieve this? And today's agent, as I said, they can already see a screen. Right? And they can click and take actions in it. That part works. But if they fire off actions, what they usually do, they move on. They don't watch what happens or recover if one step didn't succeed or something goes sideways. And we need an agent that can actually work like you do, like humans work. And one example is robotics. If you just look for a moment as how robotics do it, a robot perceives what's around it and it plans what to do and then acts. So this loop here from perceiving to planning to acting, this is actually what we also would need on a screen. And it starts here really with the first word, which is perceive. The agent has to take in the screen the way you do, not scrape the code behind the page, but what's actually rendered, the layout, the state, what just changed, the work, what we're doing, and then do it. And it would also have to keep Up in real time. Think about how we as humans work together. You jump in, you react to build on top of what each other you say. And today, agents can still don't do it. What we're doing is we're sending a prompt, we're waiting, it goes away, and at one point, the agent come back and we might have to take a couple of turns, right, because what the agent come back with is not exactly what we might wanna do, so we're Sending another prompt, say, hey, go back. Do this. Do this differently. And we have this long back and forth, which we got so used to from our chatbot experience and from this rhythm taking turns. But what we actually would need, think about it, is an agent that can react while you're still working. Wouldn't that be really cool? Right? Like, at the same time, you're working. It can also come up with suggestions, can help you And there is no waiting time. So basically, an agent that perceives what you perceive and understands what you mean. We call them perception agents. So why perception agents? Why do they matter? So first, they complete the loop on computer use. Today's agents, again, they can act. They can click. They can type. They can scroll. But what they can't do well is looking at the results and whether it actually worked out. A perception agent can read the rendered screen so it can confirm its own output instead of just firing off those actions and then hoping. Second, it doesn't need an API. Your back end process. And that's important because it works off the rendered interface, it sees the same pixels and the structure you see. And most of today's software people use every day don't expose APIs at all. And then third, the input also goes the other way here. Instead of writing a long paragraph to describe what you wanna change, let's say you're working On a website and you wanna describe all the changes you wanna apply, instead of writing this really long description, wouldn't it be great if you can just point to it and say, hey. Here, this heading needs to change. Hey. Can you update this section? This is a much more precise signal and less lossy than text, and the agents can act exactly on what you marked. So this is where we started and I'm happy to share that we just recently launched the first two pieces of our perception agent harness, open source. There's two pieces. There is annotation, which you can use to tell it what you want. And then the second piece, the verification part, gives the agent the capability to check Its own work. So let me show you the first one. So here's a very quick demo on our annotation tool. This one is a Chrome extension, so it's super easy to use, and I'm gonna play here this quick video demo. So you have the extension installed, and then you can just select different elements on a screen. So this example, we're just Around the heading there, marking the section. And maybe you wanna change it. Why not? Let's change it to red. You could also select the elements on this page. You see how if I hover over, it finds the right element. You click it, you select it, and say something, maybe double the font size. And you see also how the agent here captures on the screen exactly the feedback, the location, the style elements. And it creates this complete summary, which you can then use and then give your agent to implement. So there is no Back and forth anymore because you captured exactly what you saw on screen and the agent can see the same thing. Now, let's have a very brief look at the second one, at verification. So the idea of verification is that you can describe, let's say, in this case of the web development. You can describe in a DesignMD file what your design rules are for this. And then what happens if I play this video here, the act the agent can actually check its own work against those design specs. So it will take what you defined, the colors, the components, your layout, and it turns it into those rules if you don't have it written before yet. And it does two kinds of checks then. It does So visual check, which is really cool. So everything is on brand, for example. It's the right layout. The other part is also checking user flows. So what it does there, it actually walks through this experience, through the app, for example. Depending on the tasks available, it might add a task, it might delete a task, like a real user would. So it helps you walk through those user flows as well in an automated Fashion. And then once it's done, it's writing a report which you can review. And it's gonna call out which tests passed and it's gonna tell you anything that didn't. So ultimately, you're the one that doesn't have to click through this at midnight at the end of the day because, great work, the agent already did this job for you. Now, there might not always be A screen. Right? So I talked a lot right now. I called it perception. I talked about the agencies, what do you see on a screen. But there are times in your day where you don't have a screen. Maybe you're in the office. You're walking into a meeting with a colleague. So I did a fun experiment yesterday at the conference here. So I grabbed my colleague, Giovanni, who is also here. And, actually, on the 2nd Floor, there's a great, like Little meeting booth. We found that by coincidence. So we went in there, and we had our design meeting. And the goal here is to really kind of show you how perception is so much more than just the visual part. So in this example, what we wanna show you is perception can also be listening in the room to what you're discussing. And you can see here on the picture, both of us are wearing our b devices. Big shout out to b for sponsoring these. So we're sitting there. We have our b devices that can do a transcript. They're listening to what we're saying. And then we have this design meeting, and I had a couple of great ideas how to change this website. You will see them in a in a second here. So let's have a quick look how this changed the same workflow on this website using this device. So we had the discussion. The bee did the transcript, and you can see here on the right, we're This meeting transcript right in. There is a whole detailed summary of the meeting. There is what we discussed, and then it basically captures those insights. We have them right here, and we can click apply. So what this apply button does is it sends it straight to the agent. And you can see here my crazy idea is to turn the background to yellow, turn the heading to red, and also change an emoji directly. Applied. And it also straight kicks off the verification right away, so it creates this report. And and luckily, this color scheme was apparently into in the approved rules. Otherwise, this would have, like like, you did some weird things here. But again, you could change those rules if you don't wanna have yellow backgrounds, and it will make sure, that we still adhere to those guidelines. It would flag anything that's off. So you have the judgment call if you wanna either update the design spec Because you actually like yellow or you take an action and say, no. Fix this violation. But this is really the very first step. These two pieces are the very first beginning, and we're building out the rest in the open because these patterns can only get better if more people are using them, building on top of them, breaking things. So my ask here to you is go and Try them out. They're on our GitHub repos, open source. Tell us what we're missing. Give us the feedback what you would like to see, where this should go next because, ultimately, none of us get smart alone, And that's the whole point. We wanna build AI that makes all of us smarter together. Now if you're interested in a little bit more on human agent interactions And how we see those patterns changing, I would highly recommend this podcast by my colleague, Danielle Persick. She is a cognitive scientist and runs our AGI ACI team at the lab and discusses a lot about human computer interaction patterns with experts in the industry. You can find the podcast on on a popular podcast platform. We also have more sessions this week, so check them out. We have a booth down there. ## Slides ### 00:00:21 # AI Engineer World's Fair - Open - Akamai - Datadog - Microsoft [Logos of Open, Akamai, Datadog, and Microsoft] ### 00:00:47 # Clicking was the easy part. Amazon AGI Lab © 2026 Amazon.com, Inc. or its affiliates. ### Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Large white computer mouse cursor icon with a blue shadow] ### 00:01:22 The agent can use every tool. It still can't do the **work**. ## Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Illustration of ### 00:01:45 # The agent can use every tool. It still can't do the **work**. ## Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Illustration of multiple overlapping browser windows or application interfaces] ### 00:02:20 ## We taught computers to use computers. ### Perception Agents Amazon AGI Lab ### 00:02:49 # We taught computers to use computers. Amazon AGI Lab © 2026 Amazon.com, Inc. or its affiliates. ### 00:03:19 # Perception Agents Capability, we mostly figured out. ## Reliability is the hard part. Amazon AGI Lab [Icon of a speedometer or gauge with a needle pointing to the right] ### 00:03:55 Capability, we mostly figured out. ## Reliability is the hard part. ### Perception Agents Antje Barth / Member of Technical Staff Amazon AGI Lab [Icon of a speedometer or gauge ### 00:04:15 # Coding already made the jump. Amazon AGI Lab [Icon of a bar chart showing increasing progress, topped with a circuit board and checkmark.] ### 00:04:48 # Coding already made the jump. Amazon AGI Lab [Icon depicting a bar chart with three bars of increasing height, the tallest bar topped with a circuit board element and a checkmark] ### 00:05:24 # Coding already made the jump. ## Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Icon of a bar chart with three bars, the tallest of which has a circuit board design and a checkmark on top] ### 00:05:47 # Code crossed it first because you can **verify** it. ## Perception Agents [Icon of a document or code editor with a checkmark] ### 00:06:22 # Perception Agents ## Knowledge work is messy. The real world is messy. ### But most work isn't verifiable. Amazon AGI Lab Antje Barth / Member of ### 00:06:48 Knowledge work is messy. The real world is messy. # But most work isn't verifiable. Amazon AGI Lab ### 00:07:20 # So how do **humans** handle the messy work? Amazon AGI Lab ### 00:07:53 ## We figure things out **together**. [Illustration of two people sitting at a desk, collaborating on a computer with various UI elements and a magnifying glass] ### 00:08:18 # We figure things out **together**. Amazon AGI Lab © 2026 Amazon.com, Inc. or its affiliates. ### Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Illustration of two people collaborating at a computer, pointing at elements on the screen] ### 00:08:53 # Perception Agents ## Perceive, Act, Plan Amazon AGI Lab Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Diagram illustrating a cyclical process with an ### 00:09:17 # Perception Agents Perceive Plan Act Amazon AGI Lab Antje Barth / Member of Technical Staff **Amazon AGI Lab** © 2026 Amazon.com, Inc. or its affiliates. [Diagram showing a cyclical process: an eye icon for "Perceive" leads to a computer monitor icon for "Plan", which leads to "Act", and then back to "Perceive", forming a loop.] ### 00:09:55 # Perception Agents Perceive Act Plan Amazon AGI Lab Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Diagram of a circular flow with an eye, a computer monitor, and arrows, labeled "Perceive", "Plan", and "Act"] ### 00:10:18 # It keeps up in **real time**. TODAY: TAKE TURNS IN SYNC: OVERLAP ### Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** Amazon AGI Lab [Diagram comparing sequential actions with overlapping, synchronized actions using horizontal bars] ### 00:10:54 # It keeps up in **real time**. ### TODAY: TAKE TURNS ### IN SYNC: OVERLAP Amazon AGI Lab ## Perception Agents Antje Barth / Member of ### 00:11:17 # Why perception agents? - They close the gap on computer use. - They don't need an API or back-end access. - You show what you mean, don't describe it. Amazon AGI Lab Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** ### 00:11:51 # Why perception agents? - They close the gap on computer use. - They don't need an API or back-end access. - You show what you mean, don't describe it. ### 00:12:45 # Why perception agents? - They close the gap on computer use. - They don't need an API or back-end access. - You show what you mean, don't describe it. Amazon AGI Lab ### 00:13:18 # Perception agent harness with **annotation** and **verification**. ### Amazon AGI Lab ## Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** ### 00:13:49 # Annotation ## Perception Agents [Screenshot of a podcast webpage titled "Some Podcast" with a description about researchers discussing intelligence, and a list of episode titles including "What Even Is Intelligence? (Asking ### 00:14:30 # Annotation [Screenshot of a web page (blog post titled "Some Podcast") with an annotation tool overlay showing notes like "Make it red" and "Double the font size"] ### 00:14:48 # Visual Verification ### Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Screenshot of a dark-themed application interface, possibly an IDE or terminal, showing a search bar with "i-react" and an empty "New Session" tab] ### 00:15:17 # Visual Verification ## Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Screenshot of a dark-themed application interface showing a session with a checkpoint, activated skill, and steering documents, related to UI verification.] ### 00:15:49 # Visual Verification ### taskflow-1: Add a new task - Typed "Buy groceries" into the task input - Clicked "Add Task" button - Verified: new ### 00:16:21 # Visual Verification ## Preview report.md ### taskflow-1: Add a new task - Typed "Buy groceries" into the task input - Clicked "Add Task" button ### 00:16:52 # Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Photo of a man and a woman sitting at a table with laptops, with a monitor behind them displaying "Design Meeting!". Caption: Amazon AGI Lab] ### 00:17:32 # Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** Amazon AGI Lab [Photograph of two people working on laptops, with a monitor displaying "Design Meeting!"] ### 00:17:50 # Perception Agents Antje Barth / Member of Technical Staff **Amazon AGI Lab** [Photo of a man and a woman sitting at a table with laptops, smiling at the camera. A monitor behind them displays "Design Meeting!" with a yellow dot.] Amazon AGI Lab ### 00:18:21 ### Screenshot of a "Bee Conversations" application overlaying a podcast webpage #### Redesigning UI Elements **KEY TAKEAWAYS** - Discussed UI design changes for what appears to ### 00:18:50 # Perception Agents Antje Barth / Member of Technical Staff Amazon AGI Lab | | | | |---|---|---| | Accessibility | 0 | 0 | | Project Rules | ### 00:19:18 Just the first step. ## AI that makes all of us smarter, together. Amazon AGI Lab ### 00:19:48 Just the first step. # AI that makes all of us smarter, together. Amazon AGI Lab