# The low-latency inference engineering playbook — Charles Frye — session 2026-09-24T12:30:00.000Z → 2026-09-24T13:00:00.000Z _88 transcript lines · 51 slides_ ## Transcript Hopefully, in five years from now. And, so yeah. What what do you think that we're gonna have any Frontier models that are gonna be trained in, European AI factory? This is you you can start with this one, January. And, Yeah. I think I mean, we're acting on it at Mistral. Like, we're bringing up, like, 200 megawatt of capacity next year and one gigawatts, until 2070. So and, All of it is in Europe. So, and we are training on this infrastructure. And, actually, like, the current, mister Mars are already trained on our infrastructure. So I think in five years, for sure, we'll have Frontier Mars. We're announcing already. Okay. Great. Okay. So what's the most overrated infrastructure trend in 2026? That's fine. That's fine off. This for you? This for you as well. Out of the blue, self healing everything. What? Okay. And what's the most underrated one? Yeah. I think, like, in in infrastructure, like, it depends on which, which, layer we're we're looking at. It's a tricky question. But, like, software, in a way. Yeah. I mean, if you look at a purely infrastructure level, infrastructure is may software is probably, like, so so one of the underrated one, software enabling infrastructure. Alright. That's a good answer. I'm happy with that. So with this, thank you so much, John. This is it for for for this chat. And, yeah. Thanks, Adolf, and thanks, Jan. Please let's give it up Hello, Jan. Oops. Alright. Well, the audience is part of the five years model exercise. Right? Absolutely. Yes. We're all gonna be coming back in five years. Thank you so much. Alright. Let's give it up for you, and then head off once once more, please. Okay. And with that, we're ready for our next speaker. Our next speaker comes from Moto. Please, ladies and gentlemen, please welcome to the Stage. Charles Fry. Alright. Alright. That was great intro music. I'm very pumped. I'm ready to do, like, a jazzercise or something. And you need a lot of energy for low latency inference, not just wattage, but also, grid. So it's an appropriate start. So I'm Charles. I work on inference engineering at Modal. We're a infrastructure platform. And what I'm gonna tell you about today is our playbook that we've developed for optimizing, low latency LOM inference. So but before telling you How to make it good. I wanna talk about, like, what low latency in... What low latency inference is and why it's so important. So there's this thing that got released recently called Jev, that probably hasn't been brought up every five minutes, today. So, JEV is a decision model, a generic zero shot classifier that works really fast. And people have gotten very excited about it. People have built these really cool demos. One of my favorite from Twitter is you, like, type a word and you get a pallet associated with it. And the, the person who created this demo, Matt Deloria, said, oh, this is very cheap and fast. It feels like a leap forward for creating new UX and UI paradigms. And the response, I think, for a lot of people who've been in this field in machine learning and natural language processing for a long time was actually kinda different. It was like, oh, I'm surprised that everyone's so excited about this. And the sort of, like, base The answer here is that they made this intelligence that was always there cheaper and faster at the same time. And all of a sudden, that unlocked new use cases. If you aren't sweating that, like, you might run up to a usage limit or you might, like, spend a thousand dollars just trying to make an icon spin, and you won't wait thirty minutes for that icon to start spinning, then, like, all of a sudden, there are more opportunities available to you. So Oh, I forgot my corporate sponsorship statement. Yeah. This is, application, Jev and TypeSafe building on top of our cloud platform. So this is the kind of thing, that I'd like to help you figure out how to build yourself. And so Jev sort of sits in, like, one of three basic categories for the kinds of things that people build with, with language model inference, with sequence inference, and my sort of, like, infra brained. Thinking on the back end here, is that you kinda wanna divide these out into what is the kind of, like, figure of merit for their performance. The performance is so critical here. It's what unlocks these u UI, UX, paradigms that something like, you know, Jev for the original release of chat GPT was able to just, like, open up that was not there before. So the this performance is super critical. And for chatbots, the sort of figure of merit is how many tokens per second can you get back to a user. So tokens per second per user, for a decision model or a zero. And then the sort of final category, the sort of high latency inference category, these, like, data processors that might operate on a whole bunch of data from your back end or might consume in reducto's case, like, you know, thousands, millions of PDFs. They have a different sort of figure of merit, a different performance target. The number of millions of tokens that you can that you can process for a dollar or maybe billions. Tokens. So this is higher latency inference, so we won't talk about it. We're gonna focus on the low latency inference. So in the sort of agent category, we've been working on serving the, Kimmy models from BoomShot AI. And the, like, the difference between what you get from a sort of, like, off the shelf, Let me take the description in SG Lang's cookbook or veal and cookbook and serve it versus what you can get if you sit down and look at the. Workload and optimize, do a lot of configuration tuning, do a little bit of, like, performance engineering. You can get, in our case, we were able to get about, you know, five and a half times more, efficiency tokens per minute per GPU, but also three about three times faster per user. So, about, like, 400 tokens per second per user instead of, a little over, like, a 130 or so. And this has allowed us to, like Serve this at large scale through a variety of platforms like open router and in partnership with our customers. And then on the low latency side, obviously, you've mentioned that Jeff is running on modal, but also, we've written about, how we worked with Decagon to get the, like, lowest possible latency for their, like, for their agent operating procedures that sort of do, this decision work, inside of their voice agent pipelines. So let's talk about how that's actually done. So the sort of, like, playbook we've developed. So the first step is that you need to understand the workload that you are serving and the hardware that it runs on. And then once you have a clear picture what that is, then you can start optimizing the performance. The workload tells you what you have to do, and the hardware is what sort of does it for you. And it it says, this thing will be easy. This thing will be hard. This thing will be 10 times Fast to this other thing. So in understanding the workload, just to make sure we like, I think a lot of people have been substantively consuming AI through APIs and not necessarily thinking about what's going on inside. So just to make sure we're all on the same page. We like, you send in something like this, thou shalt not create, into a language model inference server. This, the, internal representation of the model is calculated on this. Inputs in the pre fill phase, that generates a few tokens, and then you iterate that over and over again in the decode phase to generate the outputs and complete the commandment, from the Orange Catholic Bible that thou shall not create a machine in the likeness of a human mind. And along the way, you store those internal representations in something called the kv cache so that you can, retrieve it later instead of having to recalculate it. And that's a pretty important Piece, especially if you want to serve low latency agent inference. If you're serving something that looks a little bit more like JEV, this is less important because of the source structure of the workload, the structure of the request that you're gonna receive. So with an agent, you see something like this where you have the the first user input comes in, the LMM responds, and then later you get another request from the same user that has that LMM response and their first input in it. And then you get another, maybe a tool From the l m. And once that tool call comes back, that also is going to have the whole prior context in it. And so you have this iterative construction of the input sequence turn after turn. So some things about this, like, this field and about the way LMS and artificial intelligence work are kind of like flash in the pan. But this actually kind of feels very deep and seems like something that'll stick around for a while. Essentially, the user, the model, and External systems that it's accessing are iteratively constructing this context, and they're creating something of increasing value over time. And so we should expect these kinds of interactive, like constructions to be part of how these sequence models end up working even once they become, you know, something inside of a robot moving around your house or, once they start taking in videos and outputting videos, they'll still probably look quite a bit like this. Yeah. Now for a little something that's a little bit more narrow in scope or momentary. The good news is that right now, if you're focused on high interactivity, low latency inference, you actually can usually run on no more than eight data center GPUs. So if you check out the really excellent benchmarking work from semi analysis and their inference x or inference max benchmarks, this is maybe a little hard to see, because they're very dense charts. This is, like, the least dense version I could pull out. You could See that if you want the highest throughput per chip, you want these big, like, megawatt scale, not quite a megawatt, these really huge machines that have a bunch of GPUs in a in a tight domain with each other, an NVLink domain, you need, like, what it says here, four d g of those GPUs, 64 of those GPUs. But if you wanna serve at the highest interactivity, if you want the highest tokens per second per user, you actually wanna be all the way down here. And if you look at the points that are down here, you can see Or, quantization, reducing the precision is another big speed up, has quality implications. Once you get those big wins out of the way, there's, like, kind of a grind of medium sized tasks on the host side, like, sort of regular CPU systems performance engineering stuff you gotta do, that you wins 50%, maybe as large as two x. The weirder the thing you're doing is the more wins that are probably gonna be on the host side. And then only then do you spend your time, you know, working on GPU kernels and the sort of thing that, like, everybody gets most excited about. So let's go through that real quick at a high level. Like, what do all these things look like and and when should you use them? So big wins first. Speculative decoding takes that decode process that we talked about where you generate, an output sentence like the l m is gonna generate from model is a to model is a serverless computing platform. The thing you do, instead of running this guy one step at a time, Is you take something much cheaper to run and you run it, and generate the output tokens. And you take a guess at what those output tokens might be, you speculate. And the idea here is the same as speculative execution inside processors. If you have some, like, Slack in a system, which is what happens when you run LOMs one token at a time, then you can, sort of use some of that Slack to do work that might not be accepted. So run instructions that Might not actually be on the code path inspected of execution or process tokens that might not show up in the output, like the word company here. But you like, it's such a big win to be able to run these things in parallel, especially when you're targeting high interactivity that this, is, like, very frequently a humongous win. And it's sort of linear in how many tokens you can get the speculator to guess. This is from one of our blog posts about speculators. Basically, as you increase the number of Tokens that are accepted on average, your speed up goes up as well. And these are these are integral numbers, like four x faster, eight x faster. If you're, you know, do performance work, you're used to, like, you know, sweating over 5% or 10%, but these are, like, 500%. And then one of my favorite things about this as somebody who worked on machine learning training from the beginning is that the difference between a point like this and a a speculator and model pair like this and one like this is that You, you customize to the specific workload that you want the the the speculator, or the target model to work on. You train the speculator to know how that model behaves on that data. And all of a sudden, you go from six times faster with a three times faster to five times faster. And this is, you know, the bitter lesson of machine learning, but applied to performance engineering, which is exciting. People often ask, like, wait. How can some dumber model guess what the ARTER model is going to say. There's a couple reasons why this works. The biggest one is that there's a lot of structure in outputs. And so if you would take something from your coding agent sessions like the, the thing above, and then you translate it into what the model sees. This becomes even more obvious. You have chat templating that has all these special tokens, which are usually very predictable. And then this right here is a quote from the user, which probably appears earlier in the transcript. So it's easy to Yes. There are only so many tool calls. There's only a few of them that are something you would say you would do after saying you were gonna read that file. And so this is, like, pretty easy to predict. But what the model the target model is doing is actually, like, processing these tokens and creating this rich internal representation that is gonna be used for many tokens in the future. So you by being able to, like, guess what those specific values are with a good rate, you're able to parallelize that, construction of their representation, Cool. And that representation actually gets reused inside of current speculator models. They don't just have to they're not like a Markov model, an engram speculator, or another language model from scratch. They take those rich internal representations from the target model, and they, they use that. They use what's what's going on inside the models. They're able to kind of, like, bootstrap from that. So the specific architecture there was d flash. That's the the sort of speculator architecture that we found, works best. One that we've invested the most in and gives us, like, large speed ups over both built in speculative decoding in certain models and, of course, over the baseline. After, speculative decoding and getting that right, the next big win is quantization. So with quantization, you take the floating point representation of the weights or activations of the model and you decrease the bit width. So it goes from 16 bits to eight bits to maybe even four bits. So this is a linear reduction in Marie Bandwidth Demand and a linear increase in floating point operations per second, because the the the bits are smaller, or the the the numbers are smaller. So this... When I say this requires full stack coordination, because you have to go go down to the hardware layer and be like, okay. Am I running on a GPU that can actually make use of these things? Do I have a h 100 or later for f p eight? Do I have a b 200 or later for, f p four? But it also changes model behavior. The magic of spec of decoding is that it changes nothing about the problem. Distribution of the model's outputs. But quantization does. And so you kind of have to go all the way back up to the application layer and ask, is this still sufficient intelligence? Is this you know, are the capabilities still what I need if I quantize the model? So with quantization, there's this nice little visualizer on our LM engineers almanac that shows you, like, the effect of quantization, in this case, four bit quantization on this, image of a parrot here versus the original. I think maybe, as a demo of when Quanta Station does work. I think the people in the front row can probably see that this is a little bit, a little jank. But the people in the back row, maybe, you know, people squinting at their phones, like, maybe can't see that that's, that that's been quantized. And so that's the effect that you can rely on, sort of depends on what you're doing. So these are your big wins. Let's talk about the host performance grind. So the key thing here is that the GPU does all the work, but the CPU can get in the way. The CPU is coordinating all the work. What's going on in the GPU? And you wanna make sure that the CPU work that is going on does not block the GPU work. Like, this is finished. Oh, now I need to do some work to decide what to do next, and then, oh, I'm gonna start running GPU stuff again. That's really bad. It's a bubble or a stall. If you've worked with an event loop, then you're familiar with this idea, like, don't block the fast thing. There's a key technique for this called CUDA graph capture that allows you to take, like, instead of launching, like, one GPU kernel at a time, you can just Record all the kernels that you ran one time and then just launch them all at once, which is a pretty effective way to, like, cut a lot of this host work out. One tool for keeping track of this is actually my my favorite is to just look at the power usage and the temperature of the GPUs. If you aren't close to a 100% power, you probably aren't using the whole GPU. They're very much power limited, when they're achieving peak performance. And so in this case, we are able, like, just loop by eye. Noticing that this thing was way below its total GPU power usage even at this, between this one, which is at well over 2,000 watts, and this one, which is, below it or kind of capped at it. That doesn't tell you what the problem is. You wanna use something like a profiler to get in there. You want these traces. It's a very empirical discipline. And so the Nsight systems or Torch Profiler, the tools that come from NVIDIA for this are kind of your, your go to. And if you taking a look at that Specific case, we're able to determine that one of these GPUs is actually falling behind. All these guys here are all doing the same thing, the the highlighted blocks. They're all running this DeepGem, and these three GPUs are all waiting on this one that's slow. Turned out to be like a Numa region awareness thing, that was causing one of them to always be a little slower because it was on the wrong socket. There's a lot of host level things that you can use, pie spy, like, this. Running in Python, so you could look at what's going on in Python and figure things out pretty well. We have a blog post that talks about one case in particular where we took this, like, repeated sort of allocation and turned it into just a cache. A Python dictionary. But I guess you get to call it a CUDA IPC pool handle cache if it's, AI GPU stuff. But a Python dictionary that just held onto handles for these, and suddenly that made the thing run 10% faster. So, this was mostly focused on Like, a one single replica, like, a couple of GPUs at a time. When you go to scale up, you're gonna run into routing problems. So you might predict performance before scaling that that just doesn't, happen once you scale up. So one particular thing that might happen is you might, end up with, like, tail latencies that are much higher than you expect and that kind of show up seemingly at random. And the answer is that they are kind of at random. They come from the way that you do your routing. And this is Important for agent workloads because of the kv cache. If you're not routing carefully, like, with respect to the kv cache, then the difference between, like, a cache hit and a cache miss is maybe thirty seconds versus five hundred milliseconds. And so, you measured this in one of our deployments and determined that there was variety of reasons causing instead of the, you know, four or five, concurrent requests we were trying to aim for, we we saw this major over dispersion of requests. And the solution was just to make The router smarter, had to become much more stateful, and aware application aware of things like the load and the kv cache state, but that was able to sort of pull in this, over dispersion and, massively, you know, get rid of these, spikes and massively improve performance. So then and only then are you allowed to think about, you know, GPU performance and optimizing things on GPUs. In general, kernels are relatively standard. They're close to the speed of light. Pictures are not changing that much, and so there are really good kernels out there from NVIDIA and from, like, TreeDao's group and others. So there's only a few percentage points to gain there. So that matters a lot at scale, but it's not something to worry about at the, like, you know, at the beginning of serving your own inference. The specific tool for this, Nsight Compute, allows you to, like, go all the way down to the assembler level and figure out where performance problems are coming from. But before you pull out and site compute, I'd suggest you pull out, a marker and a wipe. Board. If you look at, TreeDao's, and Together's blog about their flash tension four kernel work, the sort of the way they decided what to work on, the way they decided what problems mattered was just to write down the feeds and speeds, like, how much data is coming in, how many operations need to come, happen on it, and how quickly can we move it back out. And, that is something that you can do, you know, before, you know, from first principles. And this sort of applies across the stack if you understand. Things about the, like, hardware that you're operating with, then, you can really, you can get a lot further, without confusing yourself and wasting money. Great. So I said that we weren't gonna talk about high, about high throughput, but I actually lied. There's a little piece of work that actually is just coming out today and that I wanted to share with folks here at AI Engineer Paris. We've been thinking a lot about low latency inference at modal because of how much people You know, love coding agents, how much they love things like JEV. But there's actually quite a bit of demand for high latency inference for things that look very different. So the sort of prototypical example of this and that we're gonna talk about is AI SQL. So this is not the same as AI writing SQL, which my understanding is AI now writes all of the SQL in the world. Certainly, I wrote this, but, you know, this is this is not something I run-in production. But yeah. So this is not AI writing SQL. It's actual it's actually the other, like, direction, the opposite. It's using SQL to write prompts for AI. So the idea is, you know, you have all this information about your customers and about your products. Have you ever wanted to just ask Chad GBT, like, which of my customers might buy which of my products? And that can be expressed as, like, a relational join, the classic kind of thing that you Normally do in your database or your and in your transactional or analytic database. But the problem here is that this guy here, this prompt, is gonna, like, bankrupt you if you're running against, like, a proprietary API. So, like, running this on your database might produce, like, 8,000,000 queries all at once, and then, you know, fire them all off and there goes your token budget. Yeah. That's one way to achieve token maxing, but I think we now all agree that that's a bad idea. So, the key thing that makes this a totally different optimization problem from the low latency inference is that you have this tremendous amount of structures. This is a query plan for a little bit more complicated of a query that has a couple of joins in it. And this query plan says, these are all the sort of documents I'm gonna use to construct prompts. This is how I'm gonna have them interact with each other, and this is all the 8,000,000 or 10,000,000 or whatever prompts I'm gonna I'm gonna generate and send to Inference engine. And the existing inference engines have no idea about SQL. They, like, they only know sort of, like, request response, maybe batch request kind of semantics. So what if you took the sort of page out of the database book and you did, like, query planning and combine that with the inference engine. So we did that with, partnered with Shreyas Shankar at CMU, to get about a 14 times speed up over the, like, baseline VLM on that. Specific query that I just showed you and achieve, like, over a billion tokens per minute on a single h 100 and finish this query at just, like, $2 to execute instead of, like, almost $30 with VLM and, like, quite a bit more with, something like, even the smallest OpenAI models. So this is actually open source. We just released it today. This is the first announcement of it. You can find it at the full stack Datalabs web. Site and their GitHub. And you can read about it on our blog on, yeah, check it out here. All the information that I shared today is in greater detail on our blog and a bunch of resources about running your own inference on modal.com. I'll I'm gonna get pulled off the stage in about thirty seconds. So I'll just say the reason why I'm telling you all about this is both because I want people to run their own inference and because I think the contain the, serverless infrastructure platform that we've built at modal. Is the best place to run this stuff. So when you go to try and run your own inference, I think you'll realize the BODL is the right place to do it, whether that's, by running code yourself or by taking advantage of modal auto endpoints, our automated system for producing custom inference deployments for, for users. Alright. Thank you very much. Thank you so much, Charles. Alright. Let's give it up for Charles Osmoore. Alright. Yeah. I totally didn't don't want to run that query and bankrupt myself. That's for sure. Yeah. Our next speaker is actually gonna talk to us about the exact opposite. He's gonna talk to us about how to run models in, in smaller devices. So he's gonna talk to us about edge AI. Our next speaker is Dominic Fayak, and, he's a senior director at Arm. So please join me in welcoming to the stage. Senior director at ARM, Dominique Bayak. Alright. Hi, everyone. ## Slides ### 00:00:26 # MongoDB Engineer Paris [MongoDB logo with three green cubes] ### 00:00:49 # Engineer Paris [MongoDB logo] [c{ }p logo] ### 00:02:26 # mar # AI E # des [Logo resembling a stylized 'G' or swirl] [Logo resembling a stylized 'Q' or a person's head] ### 00:02:50 # Event Sponsors - NVIDIA - CLOUDFLARE - Daytona - Snyk - Modal - Neo4j - desc - ud [Grid of ### 00:03:25 - langfuse - Mistral - Modal - AI Engineer PARIS - neo4j - CLOUDFLARE - Daytona - AI Engineer PARIS - Qdrant ### 00:03:47 # AI Engineer Paris ## Charles Frye Member of Technical Staff at Modal Organized by Mistral STATION F, September 23-24 ### 00:04:24 # Low Latency Inference Playbook Modal [Green geometric logo for Modal] ### 00:04:49 ### Discussion about Jev and Modal #### Matt DesLauriers on Jev's understanding of color > does Jev understand colour? > incredible model, **very cheap and fast**, definitely ### 00:05:26 # Discussion on the "Jev" Model - Matt DesLauriers asks: "does Jev understand colour?" - Matt DesLauriers describes the model as "**very cheap and fast**" ### 00:05:54 [Abstract background with vertical colored blocks in green, red, orange, blue, yellow, and purple] [Small circular inset image of a person in the top right corner] ### 00:06:18 # LLM App Archetypes: Infra-Brained Edition - Chatbot+ - Prototype: ChatGPT, Claude Code - Decision Model - Prototype: Decagon, Jev - Data Processor - Prototype: Reducto, Fathom, AI-SQL ### 00:06:50 # LLM App Archetypes: Infra-Brained Edition ## Chatbot+ - Prototype: ChatGPT, Claude Code - Figure of merit: *interactivity*, aka tok/s/user ## Decision Model - Prototype: Decagon, Jev - Figure of merit: *TTLT* ## Data Processor - Prototype: Reducto, Fathom, AI-SQL - Figure of merit: *MTokens per dollar* ### 00:07:35 # LLM App Archetypes: Infra-Brained Edition - Chatbot+ - Prototype: ChatGPT, Claude Code - Figure of merit: **interactivity**, aka tok/s/user - Decision Model - Prototype: Decagon, Jev - Figure of merit: **TTLT** - Data Processor - Prototype: Reducto, Fathom, AI-SQL - Figure of merit: **MTokens per dollar** ### 00:08:07 modal.com/blog/trillion-tokens-trillion-parameters [Line graph showing Throughput (tokens/min/GPU) vs. Interactivity (tokens/s/user). Two downward sloping lines are shown, with annotations indicating "5.6x more efficient" vertically and "2.8x faster per user" horizontally.] ### 00:08:38 [Line graph showing Throughput (tokens/min/GPU) on the Y-axis versus Interactivity (tokens/s/user) on the X-axis. Two downward-sloping green lines ### 00:09:34 ## Understand the workload & the hardware ## Optimize performance ### 00:10:08 # Understand the workload. Thou shalt not create a machine in the likeness of a human mind [Flow diagram showing a "Server" component with a "prefill" step, followed by three "decode" steps. Below the decode steps is the text "a machine in the likeness of a human mind". A "KV Cache" is also part of the server. A "Client" sends input to "prefill" and receives output from the final "decode" step.] ### 00:10:35 - Thou ...t not create ### 00:11:08 # Agents are very different from "decision models". ### Caching is critical! [modal.com/blog/trillion-tokens-trillion-parameters](modal.com/blog/trillion-tokens-trillion-parameters) [Diagram illustrating agent interaction flows, showing sequences of User Input, LLM Response, LLM Tool Call, and Tool Call Result.] ### 00:11:44 # Agents are very different from "decision models". ## Caching is critical! [Flow diagram illustrating three interaction patterns between User Input, LLM Response, LLM Tool Call, and Tool Call Result] ### 00:12:03 ### Agents are very different from "decision models". ### Caching is critical! [modal.com/blog/trillion-tokens-trillion-parameters](modal.com/blog/trillion-tokens-trillion-parameters) [Diagram showing three sequences of interactions: 1) User Input to LLM Response; 2) User Input to LLM Response, then User Input to LLM Tool Call; 3) User Input to LLM Response, then User Input to LLM Tool Call, then Tool Call Result to LLM Response.] ### 00:12:37 # Good news! You can usually run on ≤8 data center GPUs. ## Kimi K2.5/2.6/2.7-Code 1T 8K / ### 00:13:05 ## Good news! You can usually run on ≤8 data center GPUs. ### Kimi K2.5/2.6/2.7 - Code 1T 8K ### 00:14:30 - Big wins - Host perf grind - Routing - GPU perf ### 00:15:07 ## Speculative decoding parallelizes decode. [modal.com/blog/achieve-sota-spec-dec](modal.com/blog/achieve-sota-spec-dec) [Diagram illustrating speculative decoding with two parallel paths: an upper path shows "Modal is a" leading to "Speculator" which then leads to "Modal is a serverless computing company"; a lower path shows "Modal is a serverless computing company" leading to "Target LLM" which then leads to "Modal is a serverless computing platform".] ### 00:15:34 # Speculative decoding parallelizes decode. - Modal is a -> Speculator -> Modal is a serverless computing company - Modal is a serverless computing company -> Target LLM -> Modal is a serverless computing platform [Flow diagram illustrating speculative decoding with a "Speculator" and "Target LLM" processing text] ### 00:16:07 # Speculation Is All You Need™ [Scatter plot showing Speedup on the y-axis against Acceptance Length on the x-axis, with a dashed diagonal line and green data points.] ### 00:16:35 [Scatter plot with a diagonal dashed line, showing green data points. The Y-axis is labeled 0, 1, 2 and the X-axis shows 0, 3, 4, with a partial label "acceptance".] ### 00:17:10 You're absolutely right! I should have read that file before I "straight yeeted it to prod". Checking now. Tool call: Read src/config.py L42- ### 00:17:38 You're absolutely right! I should have read that file before I "straight yeeted it to prod". Checking now. Tool call: Read `src/config.py` L ### 00:18:05 # "Diffusion is good" - **Target Model** - Input for `` `` `` via **Target Embedding** - **Draft Layer 1** - **KV Cache** - **Bidirectional Attention** - **MLP** - **Draft Layer 2** - **Target LM Head** - Output for **speculative decoding** `` **Legend:** - Fused Context Feature - Target Decode Token - Mask Token [Diagram illustrating a neural network architecture with components like Target Model, Target Embedding, KV Cache, Bidirectional Attention, MLP, Draft Layers, and Target LM Head, showing data flow with arrows and different colored boxes representing feature types.] ### 00:18:30 ## Quantization rules, but requires full-stack coordination. - Linear cut in memory bandwidth demand, increases FLOP/s - But this requires latest GPUs! Must check *hardware layer* - Changes model behavior! Must check *app layer* ### 00:19:02 # Quantization rules, but requires full-stack coordination. - Linear cut in memory bandwidth demand, increases FLOP/s - But this requires latest GPUs! Must check **hardware layer** - Changes model behavior! Must check **app layer** ### 00:20:04 # Quantization rules. [Two images of parrots side-by-side, comparing a visually distorted image (NVFP4) with a clear original image.] modal.com/llm-almanac/block-quants/nvidia-fp4 ### 00:20:32 # Never block the GPU. ## Before - host overhead - GPU kernel, GPU kernel, GPU kernel - CPU work -> host work blocks GPU -> GPU kernel, GPU kernel, GPU kernel -> CPU work ## After - no host overhead - GPU kernel, GPU kernel, GPU kernel, GPU kernel - CPU work -> host work does not block -> GPU kernel, GPU kernel, GPU kernel -> CPU work [modelling/host-inference-efficiency](modelling/host-inference-efficiency) [Diagram comparing two timelines: "Before - host overhead" illustrates CPU work blocking GPU kernels, while "After - no host overhead" shows CPU work running concurrently without blocking GPU kernels.] ### 00:21:03 ## Technique: CUDA Graph capture [Complex graph visualization showing parallel execution timelines and data dependencies, with many horizontal colored bars and diagonal lines] [Illustration of a GPU chip] ### 00:21:34 15 :30 0 [Line graph showing fluctuating data] ### 00:22:11 # Tool: NSight Systems/Torch Profiler (CUPTI) Straggler! [Screenshot of NSight Systems/Torch Profiler displaying a timeline of GPU operations across multiple processes, ### 00:22:35 # Tool: Just use py-spy >10% faster modal.com/blog/boosting-model-inference-performance-greater-than-10-with-a-single-python-dictionary [Diagram comparing two process flows: an initial "Tokenizer Process" leading to a "Scheduler Process" and five `_new_shared_cuda()` calls, and a second "Tokenizer Process" leading to a "Scheduler Process" with one `_new_shared_cuda()` call and an "IPC Pool Handle Cache", indicating the second flow is ">10% faster".] ### 00:23:07 # Perf after scaling might not match single-replica experiments. ### Time to first token (p50 / p90 / p95) - p50 - p90 - p95 ### Requests running vs. queued (all replicas) - running - queued .../blog/tri...s-trillion-parameters [Line graph showing time to first token (p50, p90, p95) over time, with two distinct peaks for p90 and p95] [Line graph showing requests running vs. queued over time, mirroring the peaks in the time to first token graph] ### 00:23:35 # Major cause for agent workloads: routing inefficiency. [Bar chart showing the percentage of replicas (Y-axis) versus concurrent requests (X-axis). The bars are mostly green for 2-11 concurrent requests, and red for 0-1 and 12-20+ concurrent requests.] ### 00:24:04 [Bar chart with a horizontal axis labeled with numbers 0 through 7, showing a red bar from 0 to 1 and green bars from 1 to 2 and from approximately 4 to 7.] ### 00:24:35 # Only now consider kernel optimization. - Kernels are often already close to "speed-of-light". - There are only percentage points to gain there. - Still matters at scale! But don't optimize it prematurely. ### 00:25:04 # Tool: literally just a whiteboard ## Feeds and Speeds For M=N=D=128 Feeds on B200 (per SM): 1. Tensor Cores (BF16): `5192 ops / cycle` 2. Exponential unit: `16 ops / cycle` 3. Shared Memory traffic: `128 bytes / cycle` Speeds (clock-cycles per tile): 1. Forward (2 MMAs + MN exp) 1. Tensor Cores: 1024 2. Exp: 1024 3. SMEM: 768 2. Backward (2 MMAs + MN exp): 1-CTA 1. Tensor Cores: 2560 2. Exp: ... 3. ... Takeaway: forward is bottlenecked by compute and exponential, backward is bottlenecked by shared memory bandwidth. So we overlap softmax with MMA in the forward pass and reduce shared memory traffic in the backward pass. https://triton-lang.org/blog/flashattention-4 ### 00:25:35 # ~~Low~~ # High latency inference ### 00:26:30 ```sql -- get hot leads with AI™ SELECT customers.id, products.id FROM customers JOIN products ON -- for each row in both tables AI.IF( -- run the prompt below and filter by truthiness PROMPT("{customers.profile} might buy this: {products.description}") ) ``` ### 00:27:37 # Data Processing Flow - **Second Join**: - **AI_JOIN**: neurological reaction experienced? anchor: report - Output: materialized join rows - **First Join**: ### 00:28:11 # Data Processing Flow - **Second join** - **AI_JOIN**: neurological reaction experienced? anchor: report - **First join** - **AI_JOIN ### 00:28:38 [Partial bar chart with a green bar and a label ending in "ed"] ### 00:29:06 # AI Engineer PARIS Organized by Mistral ### Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine - Charles Frye, Member of Technical Staff, ### 00:29:44 ### OpenAI API-compatible Client - LangChain - opencode - CLAUDE CODE - P - A AI - OpenClaw ```python import modal app = modal.App() ... @app.server(...) class SGLongdeepseek: ... ``` **Modal Auto Endpoints** - Modal Cloud Platform - `_modal endpoint create` **Existing Solutions** - Black Box - Create Endpoint modal.com/blog/introducing-auto-endpoints [Diagram illustrating a data flow comparison between Modal Auto Endpoints and Existing Solutions, both connecting to an OpenAI API-compatible client. Modal Auto Endpoints show a code block on the Modal Cloud Platform, while Existing Solutions are represented as a black box with a 'Create Endpoint' action.]