# Closing the Loop: An Autonomous AI Research Agent — Tim Sweeney — session 2026-07-01T20:30:00.000Z → 2026-07-01T20:50:00.000Z _81 transcript lines · 40 slides · source: full recording_ ## Transcript And I'm Paul Justine. I'm the founder and CEO of Decoding AI where I do a ton of content on courses on how to ship air products, and I'm also the co author of the Okay. Hello everyone, and thank you for attending this session. My name is Tim Sweeney, principal engineer at Weights and Biases and CoreWeave, and for the next twenty minutes, we're gonna talk about ARIA, our new AI research and iteration agent. Let's go ahead and get started. So, first off, just by way of making some noise, some clapping, who here, identifies as an ML researcher? Someone that trains models trains the brain. I heard one. Wow. Okay. Great work. Great work. What about who here is the applied engineer, the namesake of this conference? Who here actually builds the bots? Okay. Good. Expected much more. And who here is in AI management you are helping fund this compute? Okay. Okay. Nice. From the back. Lovely. Well, now that I know a little bit about you, just a little bit about me, again, my name is Tim. I have a master's in machine learning. And reinforcement learning from Georgia Tech. So I've been that, researcher. Currently building Weights and Biases agent Aria. So identify as that applied engineer. And in a previous life was the PM of Twitter's ML stack. So I hope you hopefully can connect with you middle management as well. Today's agenda is kind of broken into three sections, and hopefully each of you personas walk away with something valuable. So first we're going to learn about ARIA itself and how it can supercharge your AI and ML workflows. We're gonna Dive into auto research and see that live in a live demo in just a moment. Then we're gonna pull back the curtain and learn how we use weights and biases and, core weave to actually build ARIA. Because a lot of you in the audience are building agents yourself, and we believe a lot of these components can help you in your endeavors. And then towards the end, we'll just take a step back and identify a few key tips and tricks for making sure that you're able to productionize your systems effectively. For those of you who might not be familiar, Weights and Biases is the world's leading AI development platform. We've been in business now for nine years and have happily joined the CoreWeave family about a year ago. We have a number of products in our suite, but are really known for our models, training, inference, and Weave stack, which really helps collect data, about the AI development and machine learning workflows, and makes that information actionable, and enables users to make the best decisions about what to do next. So without further ado, let's go ahead and dive into Aria, our agent. We'll show a demo and then we'll get back to some slides. Okay. Beautiful. Let's make this a bit bigger. Holler at me if you need it to be bigger. So, what you're looking at here is a weights and biases workspace. For you, for anybody that isn't familiar, on the left hand side I actually see a list of a bunch of different experiments. In this particular project I have over 200 training jobs. And on the right hand side, I see a scatter plot of, in this case, declining metrics. Which is good. Means our loss is going down over time. And this view would be Very familiar for anyone that uses our tool. Now to ground this, we're actually, using the Carpathi auto research project, which I'm sure many of you are familiar with, but if you're not, it's just a very simple project that trains an L. L. M. And it's a great foundation for auto research type demonstrations because it's a very simple code base and allows us to improve iteratively over time. So let's jump back to the project and open up ARIA by clicking this blue button in the upper right. When I click this button, I'm, presented with the familiar Chat interface with, you know, how can I help you today? A few call to actions. And, you know, I can add different context in my project or maybe add images, etcetera. Everyone here is agent builder, so I don't need to bore you with the details of what an agent interface looks like. But let's go ahead and just, you know, enter in a basic intro here. Let's say, hello Aria. You're on stage AI World's Fair twenty twenty six. Please introduce yourself. So it's going to go ahead and chug along and hopefully emit some sort of nice emoji. Yay hee. I'm Ari. Yeah. I'm talking to the audience. Great. But now let's dive into the meat of why you came here. So I'm gonna open up this chat here, and this is a long running chat where I've been running again over 200 experiments using the auto research loop. It helped me download the code, set up my launch jobs, set up my GPU's, and is able to autonomously iterate on the code itself and the hyper parameters. We'll take a look at what it's doing in a moment, but while we're doing this, I'm gonna kick off a live iteration right here. So what I'm gonna say is Please conduct another batch of experiments. You are on stage at the AI Engineer World's Fair twenty twenty six, and we're hoping to find the best model live. I believe in you. Because we know we have to encourage our models. So it's been doing this for a while. What it what it's doing here is it's saying, okay great. I don't want to make a big architecture swing that feels a little bit too risky. So it's probably going to go for some modifications to the hyperparameters. And then it's kicking off a shell call here that is actually, executing that, Executing that experimentation loop. And we're going to check-in on this periodically throughout this presentation. But I want to help explain what's going on behind the scenes. So behind the scenes, I have set up a weights and biases launch queue. Launches are our product that allows you to connect your compute clusters and allows humans and agents to launch long running experimentation jobs, particularly by leveraging gpu's. Here I'm looking at a terminal output of my kubernetes cluster where we're actually seeing live execution All the experiments happening. So this is happening live right here. This is not a fake demo. Great. And if we jump back, we see that at this point it started the queues and now it is simply pulling and waiting for our work to be complete. So we'll jump back to that in a moment. But before, but let's dive into a few other examples. So, something else that is interesting you can do is maybe you might want to ask it something like, please summarize the highest performing runs in this project. This use case would be something like maybe a new user Or a new team member is joining your project and want to understand the research. Or maybe you've, someone's been doing some work while you were on PTO and you want to get caught up. We'll see what this comes up with in a moment. Some other pre canned, examples are finding patterns in your project. So here we can see that I asked it, hey, can you find some patterns in this research? And we see that, it identified that a new family of models emerged as the as the auto, auto research was happening. It identified that batch size seemed Seems to be a really high high, lever, parameter. It identified an architectural recipe that seemed to be quite promising. And a number of other insights that would have taken me hours or days to discover on my own, and ARIA is able to do it right for me directly in the interface that I already live. Not only is it able to emit text based, text based outputs, but it also deeply integrates with a number of weights and biases visualization utilities. So here I've actually asked it to emit a weights and biases report, which for those who aren't familiar, is essentially a markdown file on steroids. Embedded plots, charts, and and graphics. And so here, you know, it's talked about the thesis of the project. It's per it's emitted a number of data panels. And, I actually think it's quite interesting. It used, one of our more esoteric panels, the, parameter importance chart to, tell me the correlation of various different parameters within this, within this training job. It's got embedded In addition to Reports. It's also great at working with workspaces. So if you're a weights and biases user, you've spent a lot of your time, designing and working with workspaces. Well, ARIA is actually custom tuned and prompted to really understand how to build workspaces, build plots, and complement that that data analytics with real live graphics using the built in proprietary charts that Weights and Biases users know and love. So with that, let's go ahead and check back on some of our our prompts. We can see that the please summarize this project. Prompt is cooking away. It's querying weights and biases. It's applying patches. It's writing its own code. So we'll come back and check on that in a moment. And our long running training job is, still pulling for the results. We can see that we're cooking away on our GPU's. So we're frying some GPU's and doing some data science all live. And while that's cooking, let's go ahead and jump back to the presentation. We'll come back in a moment. Oh, no. We're not looking at a dictionary. We're looking at a Great. Okay. So quick recap here. What did ARIA show? What did we show in these last five minutes? First we show that, ARIA can serve as your data science companion right inside of Weights and Biases. Helping you discover insights that you wouldn't you wouldn't be able to discover as your experiments and as your team size grows. Next, we address the problem of complicated reporting and complicated plotting. Weights and Biases users are are really want to turn their insights into visual communication tools. They want to communicate with their peers and their colleagues. So Aria is built from the ground up to understand those primitives and help co pilot and drive right along right alongside in the UI. And announcing now today for the first time, we are releasing Aria on our iOS device. Or on our iOS app. So, Aria released on Monday and our IOS app, now has Aria built in. So if you're conducting hyperparameter tuning jobs, if you're training models, or if you're just researching within the weights and biases ecosystem, you can go touch grass at Yerba Buena Gardens and Your hyper parameter tuning jobs all from your mobile device. And what is this all building up to? This is building up to an a fully automated end to end research platform. Where we're not seeking to replace, RL researchers, but complement your work flows. Aria's great at orchestrating jobs, understanding GPU workloads, responding to events within the within the WANB ecosystem, and listening to researchers, looking up archive papers and collaborating on hypotheses. So we can let Aria drive the mechanics that you don't want to deal with, while you On the new ideas, new architectures, and new parameters that you wanted to try. Great. So that's ARIA in a nutshell. We're really hoping that you give it a shot, and, we'll jump back to the auto research at the end and see if we got a new best record. But before we do that, let's talk about how we use weights and biases and coreweave to actually build ARIA. So now speaking to a lot of the the AI agent builders in the room. Here's a quick market texture. On the left hand side you see that we have a web client, an IOS client that communicates with our API server. That then dumps data into our turn database and is worked on by our harness, our our worker harness. This is sort of archetypical of probably what most of you are all building in the room, and is exactly what we have on our back end. But that harness worker is a magic is a magic box. It connects to a number of important utilities. First is a sandbox where it can execute arbitrary shell calls, do do python data science, etc. And we invite you to try coreweave weights and biases sandbox to fit into your architecture. Next up, you need an LLM. Provider of course. And so if you're maybe using GLM 5.2 or one of your fine tuned models, we invite you to use, weights and biases inference and connect that to your worker as well. If you're like us, you need to run long running workloads outside of the main loop of the agent. Where you're actually training for date- for sometimes days at a time. Weights and biases launch can actually help facilitate that, and core weave gpu's can help make that compute even better. And then lastly and really most importantly, we need an observability layer. It's That your agents are able to log out their what's going on with their sessions, their turns, their tool calls, any errors that are hap that's happening, etcetera. We have a product called weights and biases weave that we log 100% of our traces to where us and our team can learn from. And that's where we move from production to offline. Where our team is able to use weights and biases weave to drive insights and identify behaviors, implement tasks with tasks which are essentially unit tests for your models, and evaluate those models in a loop. We have a Auto repository which you might choose to use weights and biases artifacts to store your agents or models. And you we omit our evaluation results to Weave, where we have a common dashboard that we can make go no go decisions on various prompt changes or architectural changes. That then feeds into a research loop which we call our improvement loop, where we form hypotheses, implement candidate agents, and analyze the evals. So we have two sort of complimentary yet adversarial research loops going on going on offline, feeding data from weights and biases weave. Ultimately to identify the best model so that we can promote that to production through our registry and close the data flywheel. So in the next just three, seven minutes or so, we'll just talk about weights and biases weave and show how we as a team use weave to facilitate this workflow. And we believe this is something that you would benefit from as well, all of you agent builders in the room. Yes. Another demo. Great. Okay. Okay. We have new responses so it's gonna be exciting when we open this up later. See if, we've got some better metrics. Okay. Let me zoom out just a little bit here. So here I'm looking at the agent dashboard. This is the live Weights and Biases agent or ARIA agent dashboard, built in Weave. Man, that is a lot of branded buzzwords there. This is the dashboard that you would get if you use our tool. And, you have, you know, span volume, conversation volume, token tracking, etc. Think of this as like a, a bird's eye view of your agent. For me, however, I I like this conversations view, which I do have preloaded in this tab. This conversations view is a live feed of all of the conversations that are going through ARIA. But it's filtered down to just the internal employees, so it's a little bit of a of a reduced set here. What I what I love is this middle spans view, which gives me a visual indicator of the topology of a trace. Different colors and and shapes indicate different things that are happening within the agent. So things like tool calls, LLM calls, thinking blocks, etc. Which really help We understand again the shape and topology of that particular conversation. I can of course open up one of these conversations and view our our conversation view where I can see the system prompt, the user message, shell calls, reasoning blocks, etc. This is where my research lead, myself, and my PM go to add notes, add feedback, add emojis, and talk about and discover those insights and those behavioral nuances we spoke about earlier so that we can turn them into tasks. Aria's built into the weights and biases system. As well. Here you'll see a summarize button, and these are sprinkled throughout the weights and biases application. I simply click summarize and we start a new chat contextualized to the thing that I'm looking at. So it it sees this and says, give me a brief summary of this particular conversation. So if you're paying attention closely, you'll realize that what we're doing is using ARIA to analyze ARIA's own conversations to then make recommendations about how to improve ARIA all within the UI. Okay. Great. While that Cooking away, I want to show you the last item, within the Weave ecosystem here. And that's signals. We've heard a lot today about the value of evals and the value of L. M. Judges. Weave actually offers an integrated L. M. Judge experience. So here if I zoom out a little bit, you'll see that I have a user frustration signal, a low quality response signal, ask user signal, etc. These are LLM judges that run live against against our live traffic. And we can see various different signals like user frustration moments or low quality responses. These help our team identify these clusters of behavior for us to go fix in next week's iteration. Let's go ahead and do a live look and see what it says. This says the user explicitly states that I'm not satisfied with the loss curve. It looks bad and it apparently that indicates frustration. So here we can see an L. M. Judge's live reasoning for why that particular flag was, indicated. Let's see. Four minutes left. Perfect. So, with that I've been using the term task a lot. And so what we're do what I've showed so far is is this live production loop where we are are are are tracing our prod logs. We're looking at them as humans. Maybe even using L. L. M. S to complement that analysis. And what we end up doing is transforming those into tasks. Now this gets a bit technical here, but our tasks are all described as yaml files. You can think of a task as essentially a unit test for your model. So here we say we have a an example user prompt that says check this run and that run. Both of these are giving good results. What can we learn from This, what's the difference? So this is an example of something we want Aria to be good at for all of you. And after the, requisite metadata, we see that we've defined an LLM judge. So here we've defined what correctness means in the context of that question. And we've then we've defined a second l l m judge that determines if the insights are actually interesting. And then we've, defined a third rule based judge that says, were you able to actually generate a result within just six tool calls? Meaning it got there with some degree of expediency. These are All then clustered together into we have about like 200 of these. They're all clustered together into an eval suite that runs nightly. And again, we use Weave to track all those evals. So here, I know it's a bit small on this screen, but what you're looking at is a listing of every night's eval. This is literally two nights ago. The evaluation for our candidate model got 73% on our production, on our eval suite against the 72% that our prod model got. Which means we're definitely going to push that forward, this Friday. And we can see a kind of a performance plot on the right. So these utilities are what you would get out of the box if you're, if you decide to pick up Weave and use this tool. Jumping back to the last conversation we had where it asked me, where we asked, can you please give a quick summary of this trace? We see that it actually analyzed the conversation, understood what the user was doing, and then ultimately decided that this was a pretty strong trace. Let's see. Got two and a half minutes left. So let's just quickly recap here. First off, what we use Weave to do is a, collect production traffic. Super critical to collect all of your production traffic so you can learn and iterate. Secondly, we use it to generate insights Both as humans as well we we do it as humans. We use ARIA and we use LLM judges to identify those behavioral nuances. We then enrich our tasks. We implement models and we evaluate using weights and biases weave as a shared dashboard where we can make decisions together as a team. That then ultimately allows us to promote the best model forward with confidence. So speaking of confident productionization, let me speak of, briefly to the managers in the room. So a few tips for being successful here. First is, invest in agent oriented observability. I'm a bit biased. I believe that weights and biases weave is the, observability platform of the future. But pick your favorite flavor. Whatever it is, log your sessions, log your turns, log your tools and feedback. This introduces an ability to catch New class of bugs in our world called behavioral bugs. Not exceptions. Not performance. But behavioral bugs. Next up, tasks and evals are the new world of CI. You've heard a lot about this. If you are a software engineer, you've written unit tests your whole life. You must develop a practice where your researchers are sitting on the same scrum team as you, developing tasks, and you're viewing the performance metrics as true go go no go decisions. But in order to complement that, you must use humans as a necessary judge. There are behavioral nuances that LLMs will not Catch. You must be using your product, and you must be manually reviewing these traces as a team at the end of the week on a board looking at the best and worst traces to understand how your model is performing. And then lastly, just maybe one more tip is to add value through context and tools. It can be really tempting to, try to over engineer the harness and do a bunch of creative stuff around memory and things like this. We found that a lot of low hanging fruit can be ascertained through simply giving your agent context about your business domain, the underlying ## Slides ### 00:00:21 # Build Your Own OS [Blurred QR code] ### 00:00:46 # AI Engineer World's Fair - WorkOS - Amazon AGI Lab - aws - #B...net - together.ai - CL... [A dark background displaying ### 00:01:18 - Amazon AGI Lab - Braintrust - together.ai - Cloudflare ### 00:01:48 # About me - MS in Machine Learning and RL from GT - Currently building W&B's Agent: ARIA - Previous life: PM of Twitter's ML Stack [W&B logo in the top right corner] ### 00:02:17 - AI Engineer World's Fair - Braintrust - Amazon AGI Lab - ver.ai [Braintrust logo, a hashtag-like symbol] [Partially visible ### 00:02:48 # Weights & Biases: The AI Developer Platform - **Models** Build and manage AI models - **Training** Fine-tune AI models on agentic tasks - **Inference** Serve hosted & fine-tuned AI models - **Weave** Iterate, evaluate and monitor agents ## Registry Datasets | Models | Prompts | Code | Metadata ## Core Reports | Automations | SDK | Skills and MCP server for agents ## Secure deployment SaaS | Dedicated | Customer-managed [Logos of AWS, CoreWeave, Google Cloud, and Microsoft Azure] ### 00:03:18 # Engineering the future of AI ### Screenshot of Weights & Biases dashboard [Screenshot of a Weights & Biases dashboard showing "Tsweeney's workspace" with a list of experiment runs ### 00:03:50 # Engineering the future of AI ### GitHub Repository: karpathy / autoresearch AI agents running research on single-GPU nanochat training automatically [Screenshot of a GitHub repository page for 'karpathy ### 00:04:23 # AI Engineer World's Fair ## Presented by Microsoft ## OpenAI ### Workspace Application Screenshot > Hello ARIA, you are on stage at AIEngineer Worlds Fair 2 ### 00:04:50 # Engineering the future of AI ## AI Engineer World's Fair ### Screenshot of Weights & Biases (wandb) application showing experiment runs - The encore ruled out several nearby ideas: ### 00:05:22 # Engineering the future of AI ### Tsweeney's workspace Please conduct another batch of experiments. You are on stage at AIEngineer Worlds Fair 2026 ### 00:05:48 ### Screenshot of wandb.ai: aria-autoreserach-01 Launch [Screenshot of a web application dashboard showing a list of runs for `autoreserach- ### 00:06:17 # Engineering the future of AI [Screenshot of a Weights & Biases (wandb) workspace showing a list of experimental runs and a log of a "seedstorm wave" experiment, with ### 00:06:53 # Engineering the future of AI ### Tsweeney's workspace - **Runs** - NAMI - NOTES - ITEF - VAL_B ### 00:07:23 # Engineering the future of AI ### Screenshot of wandb (Weights & Biases) workspace with ARIA assistant [Screenshot of the Weights & Biases web application showing "Tsweeney' ### 00:07:51 # Engineering the future of AI ### Parameter importance with respect to val/loss [Screenshot of a web application showing a table with "Config parameter", "Importance", and "Correlation" columns ### 00:08:19 # Engineering the future of AI [Screenshot of the "AIEngineer Demo - ARIA Mission" web application dashboard, showing a list of runs, two scatter plots for validation ### 00:08:50 ### Demo ### ARIA in Action ### 00:09:24 # Introducing ARIA AI Research & Iteration Agent ### DISCOVER HIDDEN INSIGHTS - Easily query, analyze, and understand complex experimentation data from training jobs, to hyperparameter sweeps, ### 00:09:49 # PUBLIC PREVIEW ## Introducing ARIA AI Research & Iteration Agent ### DISCOVER HIDDEN INSIGHTS Easily query, analyze, and understand complex experimentation data - ### 00:10:21 # Introducing ARIA ## AI Research & Iteration Agent PUBLIC PREVIEW - **DISCOVER HIDDEN INSIGHTS** Easily query, analyze, and understand complex experimentation ### 00:10:43 ## Part 2 ### Building ARIA with W&B & CoreWeave [Two robot emojis flanking "Part 2"] [Logo in the top right corner showing "W /" and a pattern of yellow dots] ### 00:11:13 ## "Marketecture" & Data flywheel [Flowchart showing data flow from Web Client and iOS Client to AgentAPI Server, then to Turn DB, then to Harness Worker, and finally to Sandbox (W&B/CoreWeave Sandboxes)] ### 00:11:42 ## "Marketecture" & Data flywheel [Data flow diagram showing Web Client and iOS Client feeding into AgentAPI Server, which connects to Turn DB, then Harness Worker. Harness Worker branches out to Job Launch / Schedule (W&B Launch + CoreWeave GPU), Sandbox (W&B/CoreWeave Sandboxes), and LLM Provider (W&B Inference).] ### 00:12:16 # "Marketecture" & Data flywheel - **Web Client** and **iOS Client** feed into an **AgentAPI Server**. - The **AgentAPI Server** connects to a **Turn DB**. - The **Turn DB** connects to a **Harness Worker**. - The **Harness Worker** branches out to: - **Job Launch / Schedule** (W&B Launch + CoreWeave GPU) - **Sandbox** (W&B/CoreWeave Sandboxes) - **LLM Provider** (W&B Inference) - **Observability** (W&B Weave) [Data flow diagram illustrating client inputs, an API server, a database, a worker, and various output services including job scheduling, sandboxing, LLM provision, and observability.] ### 00:12:48 # "Marketecture" & Data flywheel - Web Client - iOS Client - AgentAPI Server - Turn DB - Harness Worker - Job Launch / Schedule (W&B Launch + CoreWeave GPU) - Sandbox (W&B/CoreWeave Sandboxes) - LLM Provider (W&B Inference) - Observability (W&B Weave) ## IMPROVEMENT LOOP (Model Factory) - Analyze Evals - Form Hypothesis - Implement Candidate - Model Repository (W&B Artifacts/Registry) - Evaluation Results (W&B Weave) ## EVALUATION LOOP (Task Factory) - Evaluate Models - Identify Behavior - Implement Tasks Engineering the future of AI [Flow diagram showing client inputs, an API server, database, and a harness worker connecting to job scheduling, sandboxes, LLM providers, and observability. This system feeds into an improvement loop (model factory) and an evaluation loop (task factory), both interacting with a model repository and evaluation results.] ### 00:13:17 # "Marketecture" & Data flywheel ## Main Data Flow - **Web Client** and **iOS Client** feed into **AgentAPI Server**. - **AgentAPI Server** feeds into ### 00:13:50 # Engineering the future of AI [Screenshot of a Weights & Biases (wandb) dashboard for AI agents, displaying metrics such as span volume, conversation volume, token volume, cost volume, errors ### 00:14:19 ### Conversations [Screenshot of a web application showing a table of AI agent conversations with metrics like spans, tokens, and invocations] ### 00:14:51 # Engineering the future of AI ## Agents > Conversations ### Conversation ID: `019efaf5-200f-7bc1-ad8d-e5db ### 00:15:23 # Engineering the future of AI [Screenshot of a web application showing an AI agent conversation interface] ### 00:15:51 # Engineering the future of AI ## Signals [Screenshot of a web application dashboard displaying "Signals" with filters for user frustration, low-quality responses, and deletions, showing a list of user ### 00:16:24 # AI Engineer World's Fair ## OpenAI # Engineering the future of AI ### Screenshot of a web application: Signals [Screenshot of a web application dashboard showing a list of AI agent signals ### 00:16:48 # Engineering the future of AI ### GitHub: `wandb/WandBAgentFactory` - `data/tasks/wba/run-analysis/alpaca-ft-compare-sweeps/task.yaml` ```yaml instruction: - type: text text: | check fresh-sweep-2-train and hardy-sweep-3-train here. both of these are giving good results wht can we learn from this. whats 1... - type: wandb_web_page_state url: https://wandb.ai/capecape/alpaca_ft entity_name: capecape project_name: alpaca_ft task_metadata: level: L2 categories: - t:latency ... ``` [Screenshot of a GitHub page displaying a YAML configuration file] ### 00:17:26 # Engineering the future of AI ```yaml scorers: [rubric, traj-length] pass_threshold: 0.6 rubric: - id: correct_ ### 00:17:50 # Engineering the future of AI ## AI Engineer World's Fair ### Prod Evals dashboard [Screenshot of a web application dashboard displaying "Prod Evals" with a table of ### 00:18:19 # Engineering the future of AI ## AI Engineer World's Fair Presented by Microsoft OpenAI [Screenshot of a web application showing "Prod Evals" with a table of evaluation ### 00:18:50 # Recap: 3. Enrich tasks, implement models, & evaluate [Screenshot of a Weights & Biases (wandb) 'Evaluations' dashboard displaying various metrics and example traces] ### 00:19:18 # Tips for effective productionisation ## AGENT-ORIENTED OBSERVABILITY: **W&B Weave** Both development and production environments need to log sessions, turns, tools, and user feedback. This is required to debug and analyze "behavioral" bugs. ### 00:19:51 # Tips for effective productionisation ## **AGENT-ORIENTED OBSERVABILITY: W&B Weave** Both development and production environments need to log sessions, turns, tools, and user feedback. This is required to debug and analyze “behavioral” bugs. ## **TASKS & EVALS ARE AGENT CI** A fundamental part of your team's scrum process should be clustering user sessions, distilling into discrete tasks, and running evaluations regularly in CI