# What does an AI engineer do? — Aparna Dhinakaran — session 2026-10-13T14:12:00.000Z → 2026-10-13T14:30:00.000Z _63 transcript lines · 35 slides_ ## Transcript All the same thing. Standards are all for this reason. Okay? What I call In Workflow Tooling. So In Workflow Tooling is tooling that just occurs. The programmer doesn't have to choose to run it. They don't have to understand how it runs. It just happens. If it's there, it it feels and seems free, and it is reliable. Reliability is very, very important. Like, it doesn't have a ton of false positives. It's fast. It just happens. And the more that's true, look, you can apply it to your disposable software too. If it's free and fast, who cares? Making testing easier. So this is not just about, like, hey, how do we do better testing? Look, how do you make it so that nobody has to think about how to Better testing. How do you make it so that every test is fast and deterministic without anybody having to be a deep expert in testing? There's a ton of work that you have to put into test infrastructure to make that possible. For example, one of the great dreams that I've always had as a test owner is that everybody would provide in memory fake versions of their services. And I could just run those in memory fake versions of their services. Guess what? The coding agent can do that. So you can just do that now. Right? But you have to choose to do it and make it Available to everybody. And another thing is, look, maximum strictness. You just turn the strictness up everywhere. You turn you set all your lint rules to maximum. You never compromise on anything. And it just eliminates it's very similar to standardization. It eliminates a ton of conversations. Sometimes the linter is wrong. Like, as long as the false positive rate is relatively low and doesn't have catastrophic consequences, it doesn't matter because all of a sudden, there's a bunch of future lint errors or future issues that you don't have to deal with. Like, ask anybody who's been like, oh, I suddenly want to turn on my linter in this mill Inline code base. And they're like, oh, wow. Now here's my 100,000 line change that I need to make. Right? Much easier to just do that up front and have it on for everybody. Now look, like I said, I could literally stand here and talk to you about this straight for ten hours. There is much, much more you can do here. This is just a few areas, but also a whole area of software engineering discipline that you can apply on the other axis. So this lets us now answer our question, not just it depends, but yes or no. So Does code quality even matter? For disposable software, no. For durable software, yes. And we get to answer a third and even more interesting question, which is, does it have to be expensive? No. Thank you very much. As AI takes on more of the building, what does it mean to be an engineer? Here to explore why defining success may become the essential skill, Please welcome Aparna Dina Khuran, cofounder and chief product officer at Arise AI. Hey, New York. Awesome. Well, welcome to AI Engineer. Thought, given we're at AI Engineer New York, we'd kind of revisit the question of what does an AI engineer do. I feel like every year, we have an idea of what an AI engineer does, and that definition kind of changes. And as I was putting this talk together, it was super timely because Karpatia Actually, just tweeted something this weekend where he kind of he kind of talks about this growing gap between basically folks at the frontier who are running thousands of agents and swarms of always on agents. And he he really kind of drops this funnel where 6,000,000,000 plus users have maybe touched or interfaced with an LLM. One to 2,000,000,000 are maybe like casual, free tier users. 20,000,000 are professionals who have maybe started using For more... Taking some piece of work that would have taken weeks down to to days. But really, those at the Frontier Labs are... Which is a super small percentage. It's like 5,000 users. Are the ones who are running thousands and thousands of agents at once. And as I look at this, it really is, I mean, he's kind of dropped a bit of the funnel. I mean, the first question anyone looks at this is that, where am I on this? And He's kind of drawn out a bit of a road map of how does everyone get to, in the next couple of years, one of those 5,000 users who are running thousands of agents. And this has been happening, I'd say, for a couple of decades now, where the abstraction layer has gotten higher and higher. I mean, definitely faster in the last couple of years. But we've gone from engineers writing higher level abstraction languages to, I mean, just copy pasting code using local Agents to using cloud agents to... Really, the expectation now is how how do all of us start running a team of agents or factories of agents internally and build toward that? And what this has led to is essentially, what is the then and now of what an AI engineer is? And this is this is a bit, I I think, of a of a road map for some. But I think internally, this is what we've already started to kind of push towards. As an AI Your job is no longer to just write the feature or write an AI feature. It's about how do you build the loop to write that feature. It's no longer about, you know, just reading the diff and actually doing code review. It's actually how do you set up systems and agents to do the code reviews? And how do you write evals to actually do the non deterministic, to review those non deterministic tasks? It's actually no longer even about just setting up metrics and and software kind of telemetry, but it's actually, how do Have agents go through and read the observability data? And have agents actually consume that telemetry to find patterns? And so this loop that we've kind of been talking about, which is the build, the eval, ship, you monitor, you improve it, well, the AI engineer is now actually responsible not just for doing each of those steps, but how do you build the loop that does the... That that actually does each of those steps? And at Arise, one of the things we've rolled out in our product, and I'm gonna talk about a bit of What we've learned in building this is we've built signal to actually run this loop for AI engineers. What signal does is that signal actually looks at all of the traces. It reads the telemetry. It identifies patterns in that telemetry. And it can actually put up a fix or a PR if it has access to the repo. And signal, signal's an always on agent. So this is an example of how signal starts. But it actually starts off by just reading In the traces. Super simple example, but it goes through, reads, you know, all this configurable, but on a certain volume of traces that came in in some cadence, it reads it. And then the goal is to actually identify issues from that telemetry data. It has to keep track of issues that it's previously seen. So we have to consider the memory of, did it see something before that it thought was an issue? Did something seem like an issue, but it wasn't actually captured? Now as it collects more data, it actually thinks it's much more of a critical, critical issue. And so signal actually starts off by reading the traces to identify problems. And so one of the first things that that we realized was that the future of who's actually consuming telemetry data has really changed. It's no longer about humans consuming telemetry data. It's not even about humans consuming it and agents kind of taking on the task. But it really is How do you set up your telemetry data so that agents are the primary consumers of that data? And what that does is that that observability data can become the trigger for actually running and triggering these always on agents. And so the first thing that if you're building a team of a factory of really always on agents is what triggers these agents to wake up or start? And there can be a variety of triggers. For signal, for us, it's configurable to run An ongoing cadence. So every three hours, six hours, go pull the last, you know, x amount of traces, read them, identify problems. It can also be triggered by events. So Slack messages, if some user complains about something. It can be a metric based trigger. But these are essentially triggers to kick up the agent actually starting. Once the agent actually starts, we have to spend some time figuring out, well, what is What does an agent run actually look like? For us, we actually pull in all of the context of the traces, any additional third party data. So we allow collecting to APM logs, APM metrics, But all of the third party telemetry data that we want to pull in, access to the repo, access to the actual, the access to any keys that are needed to actually run this agent. But all of this information is, put inside of a sandbox, and this is actually what's started up every time the agent wakes up. And so the agent's triggered. We build a sandbox. We We put everything that we need inside of that sandbox. There is a proxy. The only way to get information in and out of the sandbox is by this single proxy. And everything that happens inside of that sandbox is actually already getting traced. And every output, so the issues that are generated and created are all Getting saved to some kind of memory. And so this happens every single time the agent actually runs. Every time the agent finishes running, so it's sweep through all of the traces. It either found something or it didn't. There's what actually stays and what actually gets thrown away. What stays is that if it found any issues, all of the output is actually saved off in some kind of memory so that the next time an agent gets spun up, it doesn't have to start from ground zero. It has a History of what are the issues that it actually found. The agents are traced. So all of the telemetry data of what the agent called, you know, if it actually put up a PR, all of those things are actually traced. And any skills that it may have learned while actually doing a run can also get saved to the actual repo. But the things that go away is the sandbox gets torn down. Any credentials, obviously get get removed. The repo acts This is removed. And so all of the stuff that doesn't need to be saved kind of gets torn down after the run of signal. And we've designed signal in a way to actually keep this agent durable. So that means that either if the sandbox fails or if the harness fails, then it shouldn't have to start everything from scratch in the middle of the run. It should actually just be able to pick up from the step that it failed. And all of the State, so called, is kind of saved off in kind of a a memory outside of the sandbox. And so we've designed this to actually be a very durable agent, so that if it fails, it can spin it back up. You can launch a new sandbox, launch a new harness to kind of pick back up where it started. So all of this is essentially the underpinnings of how we built signal to run every single run. Agent gets triggered. Agent gets started. It loaded with all the context, the evidence. It sweeps through the telemetry data. It either finds something or it doesn't. All of that gets saved to memory. If something fails in the middle, the agent can get reset up. And we built signal to also be, you know, agnostic of the model or the harness. So today, Signal supports not just cloud code, which is what we started with, but it also supports Cursor and Codex. And the goal is to actually support all of the the major, even the open source harnesses out there. And Signal actually also supports Multiple, will support kind of multiple models. So even if you're using Cloud Code, you can pick any of the models inside the anthropic family. And this is really important to us because, one, it's an awesome way for us to evaluate the quality of the agent every time we spin it up. It helps us control costs. So we can see, as we use more powerful models, the better quality maybe the PRs are. But can we reduce the Quality of the the... You know, can we reduce the cost and actually still maintain the rigor, and maintain the quality of issues that are actually detected by signal? So this is something we deeply thought about from from first principles when we were designing signal. And this is actually a pretty good segue into what does an AI engineer need to do? Well, an AI engineer really needs to kind of learn how to write evals. And so we actually, every time signal runs, signal finds an issue, it puts up a PR. Well, there's evals that are run on top of the agent to test that PR. And an eval can be as simple as, you know, testing something as part of your development workflow. You have a data set. It can be a simple prompt with with outputs that you're testing to make sure that there's no regressions. But these evals can also... They need to be kind of aligned with human judgment. And I'd argue that this is actually probably where an AI engineer Should be spending a lot of their time as they're building the system to actually run the loop. They should be actually spending time looking at the traces with their agents, identifying what the common errors are. And there's a lot of skills. I think I was hearing Lance yesterday talk about the cloud skill to build evals. We have a skill with the Rise, evaluator to actually help you build and write evals, as well as align those evals with the human judgment. But this is probably Where the shift to defining what is good becomes really, really important for an AI engineer to actually do. And for a lot of teams I I think we speak with and we work with, their question becomes, well, how do I trust the evaluator? And do I really trust that the eval is gonna catch? Or how do I trust that it's as good as my judgment? And I'd say what we spend a lot of our own time doing for our own products is that We actually evaluate the evaluators. We spend a decent amount of time on actually building these data sets, golden fruit data sets. We run evals, verify them against the human output, and then try to align to make sure that the judge actually agrees with the human, and and they're in agreement before we actually ship out, a new evaluator. And these evals can then be used both in an offline testing As well as in an online testing manner. And this this is probably what I would say as an AI engineer is more and more of what the job is becoming. And so what I'm gonna leave you guys with is as an AI engineer, you've built this agent. You've built this agent to actually run, be always on. You run evals before you actually ship it into production. You can ship it. And then once it goes into production, the goal To actually trace everything. And tracing everything now means not just tracing the agents, but also tracing the coding agents behind the agents that built this. So you can trace cloud code. You can trace cursor. You can trace codecs. But all of these coding agents can be traced so you can understand what happened in the factory of agents that you all are building. And then these agents, all the traces can then also be evaluated online. So there's an offline evaluation when you're actually testing what your... What a new change is. But as it goes into production, these become your online evals that you're running on top of the telemetry data. This for us ends up becoming the new data that then gets pulled back into the agents to read the telemetry data, the evals, and surface up kind of problems inside of the traces. So the system really becomes How do you go from collect data, eval that data, run agents on top of it so that you can find problems and figure out where the gaps are to actually improve improve your systems? So if I go back to the question that we started this whole thing with, it's what does an AI engineer do? I mean, the gap between... I think there's a pretty big gap between what's happening in the labs today and what's happening with a lot of businesses. And some of it could just be the data inside of Businesses are a lot more messy than what is very constrained and testable inside of a lab. But AI engineers are kind of the the role to close that gap. And I think it really starts with tracing everything, tracing the agents you're actually building, running evals, so being able to learn how to test these products that we're really building. And then how do you build a system so that agents can actually improve from the telemetry that's actually being collected, so that your ## Slides ### 00:00:18 # Make Durability Dead Simple - Standardization - In-Workflow Tooling - Make Testing Easier - Be Super Strict ### 00:00:52 # Make Durability Dead Simple - Standardization - In-Workflow Tooling - Make Testing Easier - Be Super Strict ## Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:01:53 # Make Durability Dead Simple - Standardization - In-Workflow Tooling - Make Testing Easier - Be Super Strict ## Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer ### 00:02:20 # Make Durability Dead Simple - Standardization - In-Workflow Tooling - Make Testing Easier - Be Super Strict ## Does Code Quality Even Matter? Max Kanat-Alexander / Principal Software Engineer CapitalOne ### 00:02:53 # Does code quality even matter? For disposable software? **No** For durable software? **Yes** Does it have to be expensive? **No** ### 00:03:18 # AI Engineer NEW YORK PRESENTED BY Arize [Stylized graphic of city buildings] ### 00:03:54 # What does an AI engineer do? Presented by Arize Aparna Dhinakaran / Co-Founder & CPO AIE NYC · Oct 13 ### 00:04:21 # 1000x Agent Engineer - ~6B (~75%) have barely touched an LLM - ~1-2B (~20%) casual, free-tier ### 00:04:51 # 1000x Agent Engineer - ~6B (~75%) have barely touched an LLM - ~1-2B (~20%) casual, free ### 00:05:19 # How we got here AI capabilities - **1970s** - **High-level languages** - Frameworks and tests - **2022-24** - **Chat assistants** - Copy code in and out by hand - **2025** - **Local agents** - You approve each step - **2025-26** - **Cloud agents** - Work in the background - **Now** - **Factory of agents** - Thousands of agents running for hours [Timeline diagram showing the evolution of AI capabilities from high-level languages in the 1970s to a factory of agents now] ### 00:05:48 # The AI engineer, then and now | The role | Last year (2025) | Now | | :---------------- | :---------------------------------------- | :------------------------------------------------------- | | Unit of work | A feature | A system to run the loop | | Who builds | You, with autocomplete and chat | A team of agents | | How you check | Read the diff | Agents doing code review, and evals on nondeterministic tasks | | Finding problems | Alerts on software metrics or user complaints | Agents read traces to surface failure clusters | | Your time | Writing and reviewing code | Defining good, reviewing verdicts | | Abstraction | Code | The factory agents write code in | ## What does an AI engineer do? Aparna Dhinakaran / Co-Founder & CPO Arize ### 00:06:23 # The AI engineer, then and now | The role | Last year (2025) | Now | |---|---|---| | Unit of work | A feature | A system to ### 00:06:47 # The AI engineering loop 1. Build 2. Eval 3. Ship 4. Monitor 5. Improve An AI engineer builds the system that runs the loop, and the evals and observability that tell you whether to trust it. [Diagram showing a 5-step process flow: Build, Eval, Ship, Monitor, Improve] ### 00:07:22 # We built Signal to run this loop - **Finds issues** - In your traces - **Shows the evidence** - The runs behind each ### 00:07:50 # Signal starts with reading traces - Says an action is done without calling a tool **High** - States balances it never looked up **High** - Pre-approves ### 00:08:21 # The future of telemetry ## PHASE 1 - Application - Observability - **Human dev** - The human is the only reader. ## PHASE 2 ### 00:08:50 # What triggers the agent Build Eval Ship Monitor Improve ### Triggers - Slack - Messages - An ongoing schedule - A metric-based trigger - A meeting or event ### 00:09:22 # What triggers the agent - Build - Eval - Ship - Monitor - Improve ### Triggers - Slack - Messages - An ongoing schedule - A metric-based ### 00:09:50 # Anatomy of an agent run - Build - Eval - Ship - Monitor - Improve ### Control Plane - **Trigger**: Cron, event, click - **Ad ### 00:10:23 # Anatomy of an agent run ## Control plane - **Trigger**: Cron, event, click - **Admit**: Quotas, caps, flags - **Compose job**: Prompt, ### 00:10:49 # What happens after each run Build Eval Ship Monitor Improve - **Run ends**: The agent finishes ### Kept, on purpose - **Output**: The answer or change it made - **Traces**: Every turn - **Memory**: Files, not chat history - **New skills**: Saved as needed ### Thrown away - **Sandbox**: Torn down after the run - **Credentials**: Short-lived, revoked - **Workspace**: Repo clone, scratch files - **Leftovers**: Caught by an idle sweep Agent ☐ Kept ☐ Thrown away --- ### What does an AI engineer do? Aparna Dhinakaran / Co-Founder & CPO arize [Flowchart illustrating what is kept and what is thrown away after an agent run, categorized into "Kept, on purpose" and "Thrown away"] ### 00:11:26 # What happens after each run - **Run ends**: The agent finishes ### Kept, on purpose - **Output**: The answer or change it made - **Traces**: Every turn ### 00:11:48 # How to keep the agent durable - Build - Eval - Ship - Monitor - Improve [Flow diagram illustrating agent durability: If a "Sandbox and harness fails" at Step 3, the process "resumes" in a "New sandbox and harness starts" at Step 3, then continues to Step 4 and Step 5. A "Saved state, outside the sandbox" stores "Each finished step, the conversation, memory, and which step is next," being "Saved as each step finishes" and "Loaded by the new sandbox."] ### 00:12:16 # How to keep the agent durable - Build - Eval - Ship - Monitor - Improve ## Sandbox and harness fails - Step 1 - Step 2 - **X Step 3** ## New sandbox and harness starts - Step 3 - Step 4 - Step 5 ## Saved state, outside the sandbox Each finished step, the conversation, memory, and which step is next [Flow diagram illustrating a process where a sandbox fails at Step 3, the state is saved externally, and then a new sandbox starts, loading the saved state to resume from Step 3.] ### 00:12:50 # Build model and harness agnostic Build > Eval > Ship > Monitor > Improve - **Eval set** - Scores every pair - **Stays put** - ** ### 00:13:17 # Build model and harness agnostic Build > Eval > Ship > Monitor > Improve - **Eval set** - Scores every pair - **Stays put** - **Knowledge ### 00:13:55 # Build model and harness agnostic - **Eval set** Scores every pair - **Stays put** - **Knowledge in files** Skills, memory, ### 00:14:18 # From assertions to evals ## Dataset - Cancel my subscription - Done. It ends on Nov 1. - Refund for order 4412? - Refunds ### 00:14:53 # Creating an eval Build **Eval** Ship Monitor Improve 1. **Look at traces** Read real runs, not just metrics 2. **Find errors** ### 00:15:23 # You evaluate the evaluator - **Case: Cancel my plan** - Human: correct - Judge: correct - **Case: Refund for #4412?** - Human: incorrect - Judge: incorrect - **Case: Return policy?** - Human: correct - Judge: correct - **Case: Change my address** ### 00:15:56 # You evaluate the evaluator - Build - **Eval** - Ship - Monitor - Improve | Case | Human | Judge | Agreement | | :------------------ ### 00:16:18 # Trace everything ## Trace your coding agents The agents that write your code ## Trace the agents you build The agents your users talk to ### Screenshot of a trace monitoring tool Trace: `291f4f7...00da6d2b` ### What does an AI engineer do? A ### 00:16:51 # Trace everything - **Trace your coding agents** The agents that write your code - **Trace the agents you build** The agents your users talk to [Screenshot of a monitoring tool showing a detailed trace of operations, including "ROUTER-AQL" and multiple "Orchestrator - RunIteration" steps with associated timings and token counts.] ### 00:17:19 # Online evaluation 1. Live traces from every agent run 2. **Evals score every span** 3. Failures cluster into patterns 4. Patterns become eval cases 5. ### 00:17:54 # The gap between possible and shipped. AI engineers are how we close the gap. 1. Trace everything 2. Run evals 3. Let agents improve it ## What does an AI engineer do? Aparna Dhinakaran / Co-Founder & CPO arize