# Great Agents Start with Great Infrastructure — Akshat Bubna — session 2026-10-13T18:05:00.000Z → 2026-10-13T18:23:00.000Z _19 transcript lines · 54 slides · source: full recording_ ## Transcript **[00:00:00 · 0]** Ready? So going back to the example of auto research and an agent doing a lot of training, what happens when the training run fails? You're... You want your agent to be able to easily look at the logs of the failure, reason about what happened, and do another iteration with a different set of parameters. And while training is running, you wanted to be able to look at the metrics. **[00:00:23 · 0]** You want to look at GPU utilization and make sure it's not stalled. Maybe you want to look at the budget and make sure you're not spending too much in your sandbox. And obviously, all of this is really important for humans. But as agents are driving a lot more compute, it's worth spending a lot of time exposing all of this to the agents themselves so they can reason about all of this. Alright. **[00:00:47 · 0]** I think, we know what we need. Why don't we have Astra build us an agent platform? Well, I don't think it's a good idea. I don't think you should vibe code a sandbox platform. And and there's three main reasons for that. **[00:01:01 · 0]** Number one is reliability. I think we're all painfully aware how hard it is to get right. Certain websites we use every day keep going down. And the reason for that is even with AI, it's actually really hard to reason about the feedback loop of, like, what happens in the case of this tail event when the system breaks. And this is why you need to be very careful in building systems. **[00:01:27 · 0]** Security is another one. You could vibe code a sandbox platform, but who is going to do all the work to like continually reason about the security boundaries and make sure it's, defensible in in an age when everything is evolving very fast. And finally scale. We have a blog post about how we scale to a million sandboxes which involves a lot of very interesting engineering design and reasoning about systems. And this is again something that you kind of don't want to leave up to a vibe code or platform because when you get to scale, you want it to work. **[00:02:04 · 0]** And on that point, we're actually seeing massive scale for a lot of things I mentioned. So, for example, for the snapshots I mentioned, it's growing quite fast. We're seeing 20,000,000 snapshots a week. We're processing about eight petabytes of data a day for for each of these snapshots. And our our users are creating almost 2,600,000 volumes a week. **[00:02:31 · 0]** And all of this is growing super fast. And just to take a customer workload for personal assistant, you need to create 10,000 sandboxes a second. You need to run, almost a million sandboxes. And they're not gonna last very long. They'll just last for, like, two or three minutes. **[00:02:51 · 0]** But when they come up, you want them to be interactive in less than a second. That's that's what this workload really demands. And the other interesting thing on scale is you actually cannot get all the scale from a single cloud platform. The world is actually getting crunched on CPUs as well, not just GPUs. So one thing we spend a lot of time doing is actually aggregating cloud capacity from a bunch of hyperscalers and neo clouds. **[00:03:19 · 0]** So our users have access to CPU capacity across the world when they need it, but we cannot rely on one cloud provider to do that. Okay. So where is all of this going? I hinted at this earlier, but the harnesses are really good at making use of dynamic environments. And what this is leading to is a lot of separation of the brains, which is the harness, from the hands, which is the sandbox of the runtime where stuff is running. **[00:03:50 · 0]** And, what this means, I have an example here, is the harnesses are going to be able to spin up environments based on what they need right now. They're good at snapshotting them and keeping them around for as artifacts that they need later. And again adjusting all of this as necessary. So I think this points at an interesting evolution of the pets to cattle analogy we're all familiar with. To recap, in the olden days, people thought of services like pets. **[00:04:26 · 0]** Each one was bespoke. You had to manage it. You had to reason about this individual deployment, specifically. And then over time, it, with containerization and Kubernetes, people started to think about it as homogeneous templates of these servers. They're all the same. **[00:04:46 · 0]** You can scale them up and down. If one of them fails, it's not an issue because our system can heal from that. And I think this is where a lot of people deploying agents are today. They're kind of using these homogeneous environments and scaling up these agents. But but I think that's really limiting, because the models are so good at reasoning about dynamic environments, snapshotting, creating lots of environments when needed and working across them. **[00:05:13 · 0]** So what what does the new world look like? One analogy here is it kind of looks like an ocean of microbes of all these organisms that are self evolving and working with each other. And credits this fix for the analogy here. But one thing I'm really excited about is as you give agents these really rock solid primitives for compute and storage and networking, they will be able to actually do a lot more, by doing the stuff I talked about, like spinning up lots of environments, working with each other, and being very dynamic. So, yeah, I and us at model really think that great primitives let your agents do more. **[00:05:58 · 0]** And, yeah, if you're interested in talking about this more, reach out to us. Thank you. **[00:16:27 · 1]** Hey, everyone. I'm Ryan. I'm the CEO and one of the co founders of Valve's AI. I'm going to talk a little bit about going from beyond the Model Card, how evaluations can be used to measure this jagged frontier, a special focus on finance for this audience as well. Yeah. **[00:16:43 · 1]** So first off, talking about Model Cards themselves, we've built some of the most important benchmarks over the last couple of years that the major foundation model labs have included in their model releases. So people I think are familiar with artifacts like this. Last week we have Sundar talking about our Vals Index on stage, and then even just yesterday MUSE models was citing our finance agent benchmark. It keeps our lives interesting to have to keep re updating these slides as new models come out. I had to double check level numbers. **[00:17:09 · 1]** You can get a sense of how capable a model is for valuable work. But that's actually driving value for a lot of companies. And I think two interesting stats that jump out at me, the groups that are able to actually track AI cost and spend are much, much more likely to are inferior proof. But we also have our fun... The x stack marks, again, it's not immediately obvious which is number one or what will be consistently two or three. **[00:17:30 · 1]** In the x axis, you've got cost. In particular, it's cost per task, not just dokka.net excel. You've got cost. In particular, it's cost per task, not just dokka.net excel models that are core to the results, maybe intuitive, unintuitive. When we actually build benchmarks, one of the most important things that we need to look at is the metric we use to report accuracy. **[00:17:48 · 1]** And so you have a different takeaway if you pick a different metric. A lot of benchmarks deal with partial credit. And so what this is answering is, in expectation, if I were to give a question or a task to a model, what percentage of the ## Slides ### 00:00:00–00:00:20 # Agents (and humans) need great observability - **AGENT** - CREATE GPU SANDBOX - START MODEL TRAINING - **TRAINING FAILED** ### 00:00:20–00:00:40 # Agents (and humans) need great observability ## Agent Workflow Diagram - **AGENT** - CREATE GPU SANDBOX - START MODEL TRAINING - TRAINING FAILED - ### 00:00:40–00:01:00 # Agents (and humans) need great observability - AGENT - CREATE GPU SANDBOX - START MODEL TRAINING - **TRAINING FAILED** - READ ### 00:01:00–00:01:20 # Why not vibecode your sandboxes?` ### RELIABILITY Last 90 days uptime: 93.09% uptime for GitHub Platform. Updated Oct 7, ### 00:01:20–00:01:40 # Why not vibecode your sandboxes? ### RELIABILITY Last 90 days uptime GitHub Platform: 93.09% uptime ### SECURITY > Sandbox ### 00:01:40–00:02:00 # Why not vibecode your sandboxes? - **RELIABILITY** ### Screenshot of GitHub Platform uptime dashboard Key stat: 93.09% uptime - ### 00:02:00–00:02:20 # Why not vibecode your sandboxes? ## RELIABILITY ### GitHub Platform Uptime Last 90 days uptime: 93.09% ## SECURITY > ### 00:02:20–00:02:40 # Agents need massive scale TODAY, AT MODAL - **~20M** SNAPSHOTS / WEEK - **~8.4 PB** SNAPSHOT DATA PROCESSED / DAY - **~2.6M** VOLUMES CREATED / WEEK ### Keynote Akshat Bubna / CTO Modal [Three charts showing growth over time for snapshots per week, snapshot data processed per day, and volumes created per week] ### 00:02:40–00:03:00 # Agents need massive scale ### ONE CUSTOMER WORKLOAD - PERSONAL ASSISTANT - **10k** SANDBOX CREATIONS / S - **800k+** CONCURRENT SANDBOXES AT PEAK - **~3 min** AVERAGE LIFETIME - **<1s** P99 TIME TO INTERACTIVE (TTI) ### Keynote Akshat Bubna / CTO Modal [Wireframe illustration of a cube with a smaller square on its top face] ### 00:03:00–00:03:20 # Agents need massive scale ## ONE CUSTOMER WORKLOAD • PERSONAL ASSISTANT - **10k** SANDBOX CREATIONS / S - **800k+** CONCURRENT SANDBOXES AT PEAK - **~3 min** AVERAGE LIFETIME - **<1s** P99 TIME TO INTERACTIVE (TTI) ### Keynote Akshat Bubna / CTO Modal [Wireframe illustration of a cube with a smaller square on its top face] ### 00:03:20–00:03:40 # You can't rely on a single cloud for CPUs anymore Keynote Akshat Bubna / CTO Modal [Map of the world composed of small illuminated dots, indicating distributed points] ### 00:03:40–00:04:00 - Harnesses are good at making use of dynamic environments - Separation of "brains" from "hands" over time [Flow diagram illustrating a system for creating and managing sandboxed environments and snapshots. ### 00:04:00–00:04:20 # Harnesses are good at making use of dynamic environments - Separation of "brains" from "hands" over time [Flow diagram showing a "HARNESS" creating "SANDBOX A ### 00:04:20–00:04:40 # Pets -> Cattle -> ??? - PETS - CATTLE [Diagram showing three distinct, complex application windows, representing "Pets"] [Diagram showing an arrow pointing right] ### 00:04:40–00:05:00 # Pets -> Cattle -> ??? - PETS - CATTLE [Diagram showing a large box labeled "PETS" containing three distinct application windows, transforming via an arrow into another large box labeled "CATTLE" filled with a grid of many identical small squares. A second arrow points to an empty space.] ### 00:05:00–00:05:20 # Pets -> Cattle -> ??? ## AI Engineer New York Presented by Arize - PETS - CATTLE ### Keynote Akshat Bubna / CTO Modal [Diagram showing a transition from "Pets" to "Cattle". The "Pets" side depicts three distinct application windows with various UI elements and graphs. The "Cattle" side shows a large grid of many identical, small, square units.] ### 00:05:20–00:05:40 # Pets -> Cattle -> Ocean of self-evolving organisms - PETS - CATTLE - OCEAN Keynote Akshat Bubna / CTO Modal [Diagram showing a progression from three distinct application modules (Pets), to a grid of many identical modules (Cattle), to a scattered collection of many small, varied, self-evolving organisms (Ocean)] ### 00:05:40–00:06:00 # Pets -> Cattle -> Ocean of self-evolving organisms - PETS - CATTLE - OCEAN ### Keynote Akshat Bubna / CTO Modal [Diagram showing three stages of evolution: "Pets" with a few distinct, complex modules; "Cattle" with a grid of many identical square units; and "Ocean" with a scattered collection of many small, varied units, connected by arrows.] ### 00:06:00–00:06:20 # Great primitives let your agents do more. [Four glowing green abstract icons: a wireframe cube, three stacked flat squares, a pyramid of steps, and a cluster of small cubes] ### 00:06:20–00:06:40 ## The **Highest-Signal** Technical AI Conferences AI Engineer hosts AI developer conferences around the world for engineers, breakout startup founders, and AI architects advancing AI engineering at the world's leading technology companies. ### Find Out What's Next Scan the QR code or visit [https://ai.engineer](https://ai.engineer) to subscribe to our newsletter and stay up to date with the latest news and events. [QR code linking to https://ai.engineer] ### 00:06:40–00:07:00 # AI Engineer Code Summit ## November 10-12, 2026 SF // CA **PRESENTED BY** Google DeepMind AIE CODE returns for its second annual code-focused summit. Catch up on the year with the world's top agentic engineers, sharing the workflows and breakthroughs changing how software gets built. **LEARN MORE AT** [HTTPS://AI.ENGINEER/CODE](HTTPS://AI.ENGINEER/CODE) **SUPPORTED BY** G2i [Nighttime cityscape with a bridge in the background] ### 00:07:00–00:07:20 # AI Engineer NEW YORK ## Stay in the Loop Scan the QR code or view the schedule at [ai.engineer/schedule](ai.engineer/schedule) [QR code linking to ai.engineer/schedule] ### 00:07:20–00:07:40 ### PRESENTED BY arize View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Logo for Arize, featuring a stylized 'A' icon in pink next to the word 'arize'] ### 00:07:40–00:08:00 # AI Engineer Europe 2027 ## ExCeL London ### 02.17 - 02.19 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. ai.engineer/europe [Black and white photos of London landmarks including the London Eye, Big Ben and the Houses of Parliament, and a street scene with classical architecture.] ### 00:08:00–00:08:20 # Stay in the Loop Scan the QR code or view the schedule at [ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to https://ai.engineer/schedule] ### 00:08:20–00:08:40 # SILVER SPONSORS - Anchor - apify - AXD CAPITAL - Chainguard - CLOUDFLARE - DE Shaw & Co - Expo - extend - FLOX - Gravitee - Judgment Labs - Kimchi - MINIMAX - NIMBLE - Pydantic - RELAI - SENTRY - specific ai - supabase - SurrealDB - tetrate - union View Full Schedule at ai.engineer/nyc/2026/schedule [Logos of various silver sponsors] ### 00:08:40–00:09:00 # Stay in the Loop Scan the QR code or view the schedule at [ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to https://ai.engineer/schedule] ### 00:09:00–00:09:20 # WE'LL BE RIGHT BACK ## AI Engineer NEW YORK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Nighttime silhouette of the New York City skyline including the Statue of Liberty] ### 00:09:20–00:09:40 # AI Engineer ## NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Nighttime silhouette of the New York City skyline with illuminated buildings and the Statue of Liberty] ### 00:09:40–00:10:00 # WE'LL BE RIGHT BACK ### AI Engineer NEW YORK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night, featuring One World Trade Center, Empire State Building, and the Statue of Liberty] ### 00:10:00–00:10:20 # WE'LL BE RIGHT BACK AI Engineer NEW YORK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night, featuring prominent buildings like One World Trade Center, the Empire State Building, and the Statue of Liberty] ### 00:10:20–00:10:40 # AI Engineer NEW YORK ## WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night] ### 00:10:40–00:11:00 # The **Highest-Signal** Technical AI Conferences AI Engineer hosts AI developer conferences around the world for engineers, breakout startup founders, and AI architects advancing AI engineering at the world's leading technology companies. ## Find Out What's Next Scan the QR code or visit [https://ai.engineer](https://ai.engineer) to subscribe to our newsletter and stay up to date with the latest news and events. [QR code linking to https://ai.engineer] ### 00:11:00–00:11:20 # AI Engineer Code Summit **Presented By** Google DeepMind **NOVEMBER 10-12, 2026** **SF // CA** AIE CODE returns for its second annual code-focused summit. Catch up on the year with the world's top agentic engineers, sharing the workflows and breakthroughs changing how software gets built. **LEARN MORE AT** [https:// ### 00:11:20–00:11:40 ### AI Engineer NEW YORK ## Stay in the Loop Scan the QR code or view the schedule at [ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to https://ai.engineer/schedule] ### 00:11:40–00:12:00 PRESENTED BY # arize View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Logo for Arize, featuring a pink upward-pointing caret with a dot below it] ### 00:12:00–00:12:20 # AI Engineer Europe 2027 ## ExCeL London ### February 17-19, 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. 3 Days - London, UK [ai.engineer/europe](ai.engineer/europe) [Black and white photos of London landmarks including Big Ben, the London Eye, and historic architecture] ### 00:12:20–00:12:40 # Stay in the Loop Scan the QR code or view the schedule at ai.engineer/schedule [QR code linking to https://ai.engineer/schedule] ### 00:12:40–00:13:00 # SILVER SPONSORS - Anchor - apify - AXD CAPITAL - Chainguard - CLOUDFLARE - DE Shaw & Co - Expo - extend - FLOX - Gravitee - Judgment Labs - Kimchi - MINIMAX - NIMBLE - Pydantic - RELAI - SENTRY - specific ai - supabase - SurrealDB - tetrate - UNION View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Logos of the silver sponsors] ### 00:13:00–00:13:20 # AI Engineer NEW YORK ## Stay in the Loop Scan the QR code or view the schedule at [ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to https://ai.engineer/schedule] ### 00:13:20–00:13:40 # WE'LL BE RIGHT BACK ## AI Engineer NEW YORK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night] ### 00:13:40–00:14:00 # AI Engineer ## NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Illustration of the New York City skyline at night with illuminated buildings and the Statue of Liberty] ### 00:14:00–00:14:20 # AI Engineer NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night, featuring prominent buildings like One World Trade Center, the Empire State Building, and the Statue of Liberty] ### 00:14:20–00:14:40 # WE'LL BE RIGHT BACK ## AI Engineer NEW YORK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night] ### 00:14:40–00:15:00 # AI Engineer ## NEW YORK # WE'LL BE RIGHT BACK View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Silhouette of the New York City skyline at night, featuring prominent buildings and the Statue of Liberty] ### 00:15:00–00:15:20 # AI Engineer ## The Highest-Signal Technical AI Conferences AI Engineer hosts AI developer conferences around the world for engineers, breakout startup founders, and AI architects advancing AI engineering at the world's leading technology companies. ### Find Out What's Next Scan the QR code or visit [https://ai.engineer](https://ai.engineer) to subscribe to our newsletter and stay up to date with the latest news and events. [QR code linking to https://ai.engineer] ### 00:15:20–00:15:40 # AI Engineer Europe 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. ### ExCeL London ### 02.17 - 02.19 2027 3 DAYS - LONDON, UK ai.engineer/europe [Black and white photo of a classical building in London] [Black and white photo of Big Ben and Westminster Bridge] [Black and white photo of the London Eye and River Thames] ### 00:15:40–00:16:00 # Stay in the Loop Scan the QR code or view the schedule at [ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to https://ai.engineer/schedule] ### 00:16:00–00:16:20 # PLATINUM SPONSORS - ORACLE AI Database - Sonar (Makers of SonarQube) - WorkOS - exa View Full Schedule at [ai.engineer/nyc/2026/schedule](ai.engineer/nyc/2026/schedule) [Logos of Oracle AI Database, Sonar, WorkOS, and exa] ### 00:16:20–00:16:40 # AI Engineer Europe 2027 ## ExCeL London ### 02.17 - 02.19 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. [ai.engineer/europe](ai.engineer/europe) [Black and white photo of the London Eye] [Black and white photo of Big Ben and the Houses of Parliament] [Black and white photo of a grand building in London] ### 00:16:40–00:17:00 # Beyond The Model Card: ## The Jagged Frontier Of AI In Finance Rayan Krishnan, CEO of Vals AI ### 00:17:00–00:17:20 # We build the finance benchmarks the labs cite ### Introducing GPT-5.5 Evaluations ### Muse Wonder and Muse Wonder Full benchmarks ### Models Claude excels across Finance Agent tasks ### 00:17:20–00:17:40 # Everyone uses AI. But what does it deliver? - Use AI regularly (McKinsey) - 89% - Scaling AI across the enterprise (McKinsey) - 44% - See any EBIT impact from AI (McKinsey) - 37% - AI initiatives that met expected ROI (IBM) - 25% - Have a real-time visibility into AI costs (KPMG) - 26% - Have established ROI from AI (KPMG) - 7% **5x** Firms with visibility into AI costs were 5x more likely to report established ROI on token spend **49%** Surveyed organizations that delayed or scaled back AI-agent rollouts because costs outweighed perceived value McKinsey State of AI 2026 (n=1,719); IBM CEO Study 2025 (2,000 CEOs); KPMG Global and US AI Pulse 2026. Different surveys: read as a ladder, not one funnel. ### Beyond the Model Card: The Jagged Frontier of AI in Finance Rayan Krishnan / Co-Founder & CEO VALS [Horizontal bar chart showing percentages for AI adoption and impact metrics] [Two text boxes highlighting key statistics] ### 00:17:40–00:18:00 # When you drill deeper, the frontier becomes even more jagged ## Pareto frontier models on the Excel Modeling Benchmark: Overall vs. LBO tasks ### Beyond the Model Card: The Jag