# What Will You Build When AI Is 10x Faster? — Natalie Serrino — session 2026-11-11T17:10:00.000Z → 2026-11-11T17:30:00.000Z _62 transcript lines · 38 slides_ ## Transcript Yes. And of course, the gold sponsors as well, which is... Without whom, again, we would not be able to be here, at least not in this venue. We'd basically be yelling out on the street corner at people, which, while not unheard of, is generally not the premier experience you try to have for something like this. Our silver sponsors also, thank you for them as well. One more round of applause for them, please. One of the hard things about dealing with the modern era In AI is it feels like this year has been about ten years in technology. We look back at what we were doing in January, and it feels like it was almost medieval times. It... The things that were best practices and made sense then, now, just seem absolutely bizarre. It's... Things get moving so quickly and it's easy to basically take a nap and feel like you've woken up and gotten, like, surprised by what happened while you were sleeping. Everything's moving faster. And that is the theme of our first talk that we're going to have because you're not here to listen to me ramble. Natalie Serino is the cofounder of Gimlet Labs and is going to talk to us now about what happens when AI becomes 10 times faster. Natalie, thank you very much. Please come on out here. Hello, everyone. It's so good to be here. Thank you for being with me this morning, first Just a bit about me. My name is Natalie. I'm a cofounder of Gimlet Labs. At Gimlet, we're building an inference cloud that's designed to produce much, much faster tokens at high efficiency, leveraging different categories and types of chips. At Gimlet, I work on our kernel generation side of things across different hardware types. Another thing about me is I love agents. I use my coding agents for everything, but I really, really hate waiting. And one thing that given that kind of surprises me is You're building an ultra fast inference cloud. People ask me this all the time actually. They're like, isn't inference already fast enough? And I'm like, what? But they have a point, right? Because I can do things and we can all do things with our agents so much faster than we could have done before. For example, this slide here, I actually made pretty using my agent and it took me with my agent much less time than it would take me to do that by myself. I'm not very good at design. And so I think that from an end to end perspective, obviously, we're getting more done. But in terms of is it fast enough, like that's still a question. So let's take a look at what happened when I use my agent to make that slide look pretty. You can see the ugly before slide on the right. So we're going to press play here. And I said to my agent, can you improve the visual look of this slide? I love Figma. I use Figma for everything. So of course, I'm going to use it. With my agents. So, what we can see on the bottom here is the timeline of what happened. And the text is a little small to see but red is inference. Red is the time spent that the model is in inference. Green is the time it's spent making tool calls and blue is the time that it spent taking my input. And we can see that, you know, this this agent is doing good work. It's doing a lot of thinking, but I'm waiting here. I'm actually waiting so long. I sped up this video nine x because I didn't want all of you to have to wait for the full turn day. And it ended up being about two minutes long, but we can see at the end of this after my one input, we have this nice pretty slide at the end. So, you know, on one hand, this is actually much faster than a human can do. But on the other hand, in those about two minutes, I have probably moved on to another task. I'm not really focused on this anymore. I've context switched. And when we look to, you know, the inevitable progression of inference getting faster and faster, I think it helps to look back in time and look back To the Internet era. I mean, we're still in the Internet but like, you know, what people canonically call the web era. And what we always see with a new shift is that with each new major wave, new capabilities are unlocked. And so we do things that weren't possible before that technology like using an agent to edit my slide. So I think we should take a step back to 2000 in the early days of the Internet. I think many people here weren't even alive, so it'll be a history lesson for you. And we're gonna look at a little application called SquirrelMail. SquirrelMail Actually open source so you can run it today. It was a web application for, you know, doing your email back then. So, on the left, what we have here is I'm running SquirrelMail and I'm running it with simulated 2,000 dial up speed and then on the right, it'll run with twenty twenty six speed. So, the speed that we're used to today. So, the both sides are going to take the same actions, but, the left side, I'm using it with that simulated lower speed internet. And we can see that, you know, it's doing its job. Back in 2000, loading an image like this would have blown people's minds, but our modern eyes are like, wow, that took forever to show up. And so, we can see these lags. We can see a noticeable thing, but on the same time, someone might have looked at this back in 2000 and said, This is fast enough. I don't actually need this to be faster. The thing is though, all of the major giants back in that Era did find out it mattered. Pretty much every major company from Amazon to Yahoo to Google to Bing, all of... Or Bing at part of Microsoft, like all of them found that even very small delays cost them people doing as many clicks. It cost them literal money. Even a hundred milliseconds of delay added on to the typical experience had a demonstrable impact on the bottom line. And even when you got rid of that delay Those same users didn't come back to the platform as often as they did without that delay. And this makes sense because fast products keep us in flow state. They provide a stickier UX. About one hundred milliseconds feels instant to us. One seconds one second of wait, I still keep my train of thought. It's not a big deal. I don't really notice it. Ten seconds, we're gonna start turning to something else from a psychology perspective. Maybe we pull out our phone, maybe switch To another agent. But today, from data from June, the median coding agent turn is sixty three seconds. Each workload has different speed requirements. We don't need the same speed for everything, but over time, our expectations do rise and the fastest days of fastest internet in the squirrel mail days would feel incredibly slow to us and we can keep we can expect that trend to continue for inference speed as well. There's a problem though and the problem is Everyone knows that we have massive compute scarcity. Today's infrastructure is under incredible pressure to deliver two things at the same time, more tokens and faster tokens. Unfortunately, these goals are fundamentally at odds because faster tokens require more energy to produce than slower tokens. So with the same energy footprint, you can't make as many of them. Back before this was a session and not a keynote, thanks to upgrading this to a keynote everyone. We're going to play a game that I made called Token Mint Factory and you can actually play this. I will not be offended if you play it right now even, but just so you know, it does not work on phones, it only works on your laptop. And basically in this game, you're running a Token Mint Factory. You're printing literal tokens and there's different modes. The modes are you can play to maximize how quickly you can put out one single token or you can maximize how many tokens you can put out in sixty seconds, or you can play the SLA mode where you have Have to maintain a twenty second SLA on each token for it to count, but you're still trying to put out as many as you can. And as you play, you'll quickly see that the strategy is extremely different for doing fast tokens versus doing slow tokens. And neither one is actually better or worse from a decision perspective. It's just a matter of what the application needs and what makes sense for your factory. So, I took real runs from people at my company that played this game and I made a Pareto frontier of these tokens in the game. We can see there's a real life Pareto frontier of this game as well. If we have faster tokens, we don't have as many and if we have more tokens, they're not as fast, but there is a way to get more up into the right and up into the right is where we wanna go. We want more faster tokens. I don't want to compromise. I want both. So, the game has a concept of adding upgrades to your equipment and what we can see is that compared to the no upgrade baseline, these technological innovations help drive us up into the right and get more tokens and Faster tokens. So how do we upgrade our real life assembly lines, so to speak, to make making our actual tokens faster? One thing that we do at Gimlet for this is we really believe that you have to use the right accelerator for the right job. And so, we actually take models and we split them up into different pieces and we run each piece on a different category of accelerator. So we might run one piece of the workload on a GPU and another piece of the Workload on something like an SRM centric accelerator. And we find that even with the network overhead, this delivers dramatically faster tokens. It's essentially like the same thing as the upgrade to the equipment in the token factory when you better match the accelerator to the job. Unfortunately, we have ended the time of free lunch transistor scaling, right? So now we have to look to other approaches to make things faster. And so with this, it's a little hard to read, but we can see the same real life Pareto frontier for a front Your model. If you look at simple homogeneous GPU inference, it's very high throughput and then the latency drops off a little bit. You can't get it as fast because the y axis is how many tokens that I can make and the x axis is how fast these tokens are for each user and we can see that depending on how you split the model up because there's different ways of doing it, you can get much faster tokens for the same power footprint. You can get many more tokens for the same speed or you can get More and faster tokens. So as an industry, we need to continue pushing, pushing, pushing on this because otherwise, our poor data centers are just going to keel over. So I just wanted to show a quick demo of this. So, using a coding agent, build the same app using heterogeneous inference that leverages different chip types versus the traditional one. So, this is going to be the game Snake and we can see on the left, the heterogeneous fast one's done. I can see in my flow state. I can keep adding things. I can keep playing. But on the right side, it's still going. And by the way, I still think the left side is too slow. We need to make that even faster. And then we can add a competitor to the game and we can see even with the tool calls, even with all the things that it's doing, it's going to complete and then the other one will still be hanging because it's fundamentally about how can you drive more efficiency and better performance with the same resources. It's gonna go on. I'm not Make you guys wait for this. One thing that people often point out too is, is inference the bottleneck? Our tool calls the bottleneck. And I think this is a great point. If we look back to the turn timeline from the time that I was spending updating my slide, we can see that 80% of it was inference and 18% of it was tool calls. If we simulate that same exact thing and have 10 x faster inference, we can see that 67% of that will be tool calls at that. Point if we don't optimize the tool calls. That's why it's important that we have to constantly attack the bottlenecks of the application and get the low hanging fruit where we can. Today, people are focused on inference speed and maybe we haven't optimized the tool call side and how tools are called as much because it's not the bottleneck. If I cut away the tool calls on the upper one, it wouldn't even really matter to the end application, but once we get much faster, we have to think about how to call these tools much more efficiently and much more performantly because otherwise, they're going to be the reason that we're all waiting around. I think the good news is that optimizing these speeds and what we're all collectively doing as an industry is a verifiable task. If I know what my inputs and outputs should be, I can have agents do this very, very full stack and that's where we're increasingly moving to. We're having self improvement loops with these agents to drive better efficiency on the fly as our inference is running. And at Gimlet, we're building recursive self optimization for this that pulls from low Telemetry from our data centers all the way up to the model layer and trying to use all of it to improve the overall efficiency of the workload because if you just optimize one small piece, it might not make that much of a dent. I wanna show another demo because I think that cybersecurity is on everyone's mind right now. We're all seeing like the huge incredible advances in AI driving awareness that there's all these vulnerabilities in our software And I think that this is an area where Faster inference can really, really help. So this is a dumb fake scenario, but it's a real application and a real demo where imagine that you're running the dog club and that's a website for dog lovers to upload and vote on pictures of dogs. Dogs. However, unfortunately, like we often see, one of your node dependencies has a vulnerability and this is a real vulnerability. This is really a node tar. And you're running a security agent on your application and it's trying to monitor Or live traffic and see is there anything going wrong that I don't know about. This agent doesn't know about this vulnerability, but it does know its job is to protect your application. So, let's do a quick turn with normal user traffic. We're going to upload a dog biscuit and on the upper side, we have the application logs and on the bottom, we have the defender and the defender is monitoring but it's not in the critical path because it's not fast enough to actually respond to every user. Quest live. It's monitoring things async. And we can see Biscuit gets uploaded and we see that this is an okay upload. The defender doesn't see any problem. But now what we're going to do is have this defender operating when there's a malicious upload and this upload is exploiting the real vulnerability where in node tar, there was a release where you could basically overwrite files that you weren't supposed to overwrite based on the destinations of what was being unzipped. So This one, we have Ranger, a fake bad, malicious dog, not really a dog. And what we're gonna see is that the defender is going to figure out that this is a malicious thing. The agent is smart enough to detect that it's doing something that it shouldn't be doing even though it hasn't seen this exact vulnerability before, but unfortunately, Kat's malicious upload has taken over, so your website's gone. It's too late. You didn't react in time. But if we retry this same exact malicious upload, but this time with an ultra fast defender, and again, this is a real a real demo fake use case. We can see that with ultra fast inference, we can react to things and do things online much more quickly and this should change the way that we think about how to use agents in the first place. We're We're all thinking about how to use them in an async way, but we should think about how to use them in online ways as well. So we can see on the bottom that the worker blocked their request and it was Like, what are you doing? This is obviously wrong. So faster Internet took us from squirrel mail to on demand high resolution streaming. Right? Like if you just look from this to this, it's crazy. If you showed this to my like five year old self or something I would have been like... My mind would have been blown. And this is just one of many things that I can select. So what does a 10,000 token per second world look like? We're at in the like low hundreds today. Does it look like when we get to a million tokens per second? Like what does that unlock for us? And I think that we should all ask ourselves this too. It's important in AI to skate to where the puck is moving, right? It's changing so fast. We need to be ready. We need to think boldly. It's not enough to just do what we've always been doing. I think we've all had the experience where we make an audacious request to our agent and after the latest model release, it just works. And it's gonna be the same thing with latency. And so I think that we can think about idea Is that are both coming soon when we hit a certain latency threshold what we can do, but also the stuff that is real life sci fi. Like if you look at something like Jarvis or something, there's no way today's inference could be fast enough to power that experience for Ironman, but we will get there. So I'm really excited about online use cases, online security, online things of like, I want to just tell my agent what I wanna build and it's building it right in front of me and I don't even have to type anything. I think we're gonna have new patterns Interaction. You know, think tab complete for everything. How can we have the user side of things be as quick and immediate as possible because the inference side is also just as fast to accommodate it. And we're going to have so many more agents. We're gonna have even maybe transformers driving robots. Right? There's so much that we can build when this gets unlocked and I think as builders on the frontier here, we should all be thinking about this too. Thank you so much for being with me, this morning. You can find me on x at n serino. We're hiring at Gimlet. We'd love to talk to you all. And then if you wanna play the token mint game, you can also find that QR here. Thank you so much, everyone. As AI models get better, are our old best practices holding us back? To rethink how we work with Cloud Code, please Please welcome Lydia Halley, member of technical staff at Anthropic. Hi everyone. Oh my microphone already works. Sorry, we have some technical issues this morning so hopefully everything Works. ## Slides ### 00:00:11 # AI Engineer Code Summit - Atlassian - Databricks - Factory - Google DeepMind - Exa - Overcut - Devin - Firecrawl - Paras ### 00:00:39 # AI Engineer Code Summit Partners - FACTORY - Google DeepMind - databricks - exa - Sonar - Vercel - Firecrawl - DATAD ### 00:01:11 # AI Engineer Code Summit ## Sponsors - Atlassian - Factory - Google D... - databricks - exa - OVERCUT - Vercel - ### 00:01:39 # AI Engineer Code Summit - Atlassian - Factory - Google Dev - Databricks - Exa - Overcut - Devin - Soc - ### 00:02:11 # AI Engineer Code Summit - Databricks - Overcut - Vercel - Keycard - [Stylized 'S' logo] [A grid of sponsor logos for ### 00:02:37 # Hi, I'm Natalie! Cofounder, Gimlet Labs [Circular photo of a person in a green hoodie with foliage in the background] ### 00:03:07 My agent helped me slide much faster and have been able to do 37° 37' 08" N 122° 22' ### 00:03:40 # "Is inference already fast enough today?" ## Agents are enabling incredible new product experiences! My agent helped me make the previous slide much faster and better than I would have been able ### 00:04:08 # About Me - Hi, I'm Natalie! - Cofounder of Gimlet Labs - Gimlet Labs is an inference cloud focused on delivering much faster tokens with high efficiency leveraging ### 00:04:39 # About Me - Hi, I'm **Natalie!** - Cofounder, Gimlet Labs DIFFERENT CHIPS - ONE INFERENCE CLOUD **GIM ### 00:05:07 With each major technology wave, new experiences and capabilities are unlocked. Let's step back in time to 2000, in the early days of the internet. [Illustration of ### 00:05:39 # 2000 dial up speed ## Email Inbox (SquirrelMail) - **Folders:** INBOX (8), Trash (purge), Sent, Notes - **Current Folder ### 00:06:07 # 2000 dial up speed - Folders - (refresh folder list) - INBOX (8) - Trash (purge) - Sent - Notes # 2026 internet speed - Folders - (refresh folder list) - INBOX (8) - Trash (purge) - Sent - Notes [Image of Earth ### 00:06:44 # In the 2000s, the industry observed that small page-load latencies had demonstrable impact on product metrics. - **Amazon 2006**: +10 ### 00:07:07 # AI Engineer Code Summit Presented by Google DeepMind # Fast products keep us in flow state, providing stickier UX. ## 0.1 s feels "instant" ### 00:07:39 # What Will You Build When AI Is 10x Faster? Each workload has different speed requirements, but over time our expectations rise. The fastest internet of the SquirrelMail days ### 00:08:12 Due to exploding demand, today's infrastructure is under massive pressure to deliver both - more tokens - faster tokens. Unfortunately, these goals are at odds, because faster tokens require more ### 00:08:35 # Let's play a game to illustrate throughput vs latency - You run a token mint factory! - Play in different modes, minting tokens quickly, minting a lot of tokens, or ### 00:09:13 # Let's play a game to illustrate throughput vs latency - You run a token mint factory! - Play in different modes, minting tokens quickly, minting a lot of ### 00:09:40 # Faster tokens = fewer overall tokens MORE TOKENS ↑ on-time tokens per 60s game avg latency per token FASTER TOKENS → ## What Will You ### 00:10:10 # How do we "upgrade" our real-life assembly lines and push inference Pareto frontiers up and to the right? ## EXAMPLE WORKLOAD SPLIT We can mix chip architectures within a single workload, dynamically splitting it and routing each phase of inference to the best available hardware. ### What Will You Build When AI Is 10x Faster? Natalie Serrino / Co-Founder gimlet labs [https://gimlet.ai/](https://gimlet.ai/) [Flow diagram showing "Prefill" leading to "Decode" leading to "Tool calls", with "GPU" under Prefill, "Specialized Accelerator" under Decode, and "Server-Side CPU" under Tool calls.] ### 00:10:38 # How do we "upgrade" our real-life assembly lines and push inference Pareto frontiers up and to the right? ## EXAMPLE WORKLOAD SPLIT - Prefill (GPU) - Decode (SPECIALIZED ACCELERATOR) - Tool calls (SERVER-SIDE CPU) We can mix chip architectures within a single workload, dynamically splitting it and routing each phase of inference to the best available hardware. ### What Will You Build When AI Is 10x Faster? Natalie Serrino / Co-Founder gimlet labs [Flow diagram illustrating a workload split into three sequential phases: Prefill, Decode, and Tool calls. Below each phase, the corresponding hardware is shown: GPU for Prefill, Specialized Accelerator for Decode, and Server-Side CPU for Tool calls.] [QR code for live translation] ### 00:11:11 # Multi-silicon inference allows us to meet workload SLAs with much higher efficiency and performance than homogeneous inference alone. ## What Will You Build When AI Is 10x Faster? ### 00:12:07 # What Will You Build When AI Is 10x Faster? ## Natalie Serrino / Co-Founder gimlet labs ### Heterogeneous inference w/ Gimlet - Ready ### 00:12:40 # AI Engineer Code Summit PRESENTED BY Google DeepMind # Big speedups in one area lead to new bottlenecks in other areas. - Inference - Tool calls - Harness ### ### 00:13:07 # AI Engineer Code Summit ## Big speedups in one area lead to new bottlenecks in other areas. ### Turn timeline (original speed) - 82% inference - 18 ### 00:13:41 # Optimizing inference speed is a verifiable task, making it a good fit for agents. At Gimlet, we are building full-stack recursive self-optimization into our cloud, so performance can ### 00:14:39 # Demo: fast inference for online exploit mitigation - Scenario: You run The Dog Club, a website for dog lovers to upload and vote on pictures of dogs. - Unbeknownst to you, your site has a critical vulnerability in a nodejs dependency! - You have a security agent deployed and monitoring traffic to your site ### 00:15:08 # Demo: fast inference for online exploit mitigation ## Scenario: - You run The Dog Club, a website for dog lovers to upload and vote on pictures of dogs. - Unbeknown ### 00:15:39 # Normal user traffic ### Top dogs this week ``` application / upload + worker run 8ba3de02 +0.000s ingress INFO POST ### 00:16:08 # Standard-latency defender + a malicious upload ## The Dog Club: Top dogs this week A file named `ranger-profile.tar` is shown as "UPLOAD ACCEPTED". ### 00:16:40 # Retry the malicious upload, with an ultrafast defender ### TheDogClub.com Top dogs this week ``` application / upload + worker +0.000 ### 00:17:08 # AI Engineer Code Summit Presented by **Google DeepMind** ### Faster internet took us from Squirrelmail to on-demand high resolution streaming! What does a 10,000 ### 00:17:39 # What new experiences will you build when AI is much faster? ## COMING SOON - A voice assistant with human-level interactivity - Fix the outage while it is still happening - ### 00:18:08 # What new experiences will you build when AI is much faster? ## COMING SOON - A voice assistant with human-level interactivity - Fix the outage while it is still happening - Block attacks immediately as requests execute ## A BIT CRAZY - Instantly build and compare 100 different UIs - Live voice to prototype - Build 3D models in seconds from a description ## REAL-LIFE SCI FI - Apps are written on the fly as we use them - Instant negotiation among hundreds of agents - Robots that dynamically collaborate with humans ## What Will You Build When AI Is 10x Faster? Natalie Serrino / Co-Founder gimlet labs [https://gimlet.ai/](https://gimlet.ai/) [Timeline graphic showing "COMING SOON", "A BIT CRAZY", and "REAL-LIFE SCI FI" categories] [QR code linking to gimlet.ai] ### 00:18:39 # What new experiences will you build when AI is much faster? ## COMING SOON - A voice assistant with human-level interactivity - Fix the outage while it is still happening - ### 00:19:07 # Thanks! You can find me on X at @nserrino ### Sign up for early access [https://gimlet.xyz/early-access](https://gim ### 00:19:40 # AI Engineer Code Summit ## Rethinking Claude Code Best Practices Lydia Hallie / Member of Technical Staff ANTHROPIC [https://www.anthropic.com/](https