# The Unreasonable Effectiveness of Separating the Task from the Model — Maxime Rivest, Isaac Miller — session 2026-07-02T16:40:00.000Z → 2026-07-02T17:00:00.000Z

_62 transcript lines · 0 slides · source: full recording_

## Transcript

And over time telling the sender, it's now time for you to send me the next chunk of data. And the receiver can delay those grants. So for example, if the receiver has 10 messages that are incoming, there's no point in sending grants to all 10 of them because then you'll just get congestion in the in the top of rep queues. So it can use the grants to reduce congestion. And then it can also use the grants to give preference to its most favorite messages, which would be the shorter ones. So it's a way of of implementing SRPT by favoring Messages. The third aspect of HOMA is that it takes advantage of the priority queues in modern switches. So modern data set of switches have more than one queue at each egress port, typically eight. And they can be used in a priority mechanism where packets get transmitted preferentially from the highest priority queue. So I've shown only two queues on the slide here, but typically there's more than that. You can specify in a packets using the various fields of the packet. You can specify which queue it should Into. And so HOMA dynamically makes those choices in a way to give priority to shorter messages. So if we go back to the in cast example from a few slides ago, all of those long message will pile up in the lowest priority queue. But if there's a short message coming, it will use a higher priority queue. And so it will immediately bypass all of the queued packets from the longer messages and get through to the destination more quickly. So how much of a difference does this make? Here's a on this slide, I've got one sample benchmark that I use as part of my tuning and evaluation of HOMA. It consists of a workload of a bunch of machines on a network that are exchanging messages back and forth of different sizes, ranging from very small to very large. And on this graph you can see on the x axis is the message length from about 50 bytes up to a megabyte. The y axis shows you the round trip time for messages of that length. So this uses request and response messages that are the same length. You can see TCP in green, HOMA in blue. And the y axis is round trip time, so lower is better. And for each protocol, I've got two curves. One curve is the p 50 curve. That's the median latency for messages of this length. And then p 99 is the ninety ninth percentile, I. E. Tail latency for messages of this length. So I want to point out two things. First, the p 99 for short messages is dramatically better for HOMA. So with TCP, it's more than a millisecond. Tail latency, HOMA is less than a hundred microseconds, about 13 times faster. Second, interestingly, you might think that because HOMA favors shorter messages, that long messages suffer and get worse performance. It turns out that's actually not the case. Even on the longest messages, Homa is almost a factor of two better than TCP. I don't have time to explain that today, but it has to do with the fact that Homa uses run to completion approaches, which are much more effective than fair than the fair scheduling. Used by TCP. So just to wrap up, the role of short messages in AI appears to be increasing. I think it's likely that it's going to continue to increase. We'll see over the next year or two if that happens. And I just want to pose a question to you. As you're running your applications and measuring performance and seeing what the bottlenecks are, ask yourself, is high latency for short messages affecting your throughput? If the answer is yes, then just know there Solution available. You should give Homa a try. You can probably reduce your tail latency by an order of magnitude or more. And by the way, this is Homa is basically my life mission right now. I'm sort of semi retired from Stanford. And the reason I did that is so I can spend 100% of my time hacking on Homa. So I'd be delighted to work with you and help you if you decide you want to experiment with Houma. If you need help getting started, answer questions, bug fixes, whatever, you know, I'd be happy to work with To try and make you successful with it. So, yeah, if that is interesting, feel free to contact me. My email's on the slide, or you can Google me too and find me over the Internet. So thanks very much for listening, and I hope to hear from some of you. Please welcome to the stage the core contributor and the lead maintainer at DSPI, Maxime Rivest and Isaac Miller. Wow. Isaac, myself, all of the DSPI community are so grateful to be here today to get to talk to you about AI program. I mean, DSPIME and the unreasonable effectiveness of separating the tasks from the model, its harness, and all of the implementation details. When you think about it, in programming, if we want to repeat the tasks often, we make it a function. We believe the same should be true for AI programs. Functions are awesome. Functions are reusable, composable, testable, and optimizable. To make a function, you give it a name, you define some inputs, some output And then you have some implementation logic inside of it. You get to reuse your functions thousands of times, you can optimize it, but you can also compose it into bigger programs. One of the really nice things about functions is that you can also package it and distribute it and someone else can use it and they just need to know about the contract on top of it to use it and they can treat it as a black box. DSPI brings all of these proper These two AI programs. And so DSPY is an open source software in Python that lets you, like I said, bring these properties to your AI workflows and AI programs and it gives you all of the toolings you need to do that. Why do you want that? Well, we have been inventing a lot of terms in our fields in the last three years. It's growing fast. We have new models Coming every other week, we have new techniques, new strategies and if you're like me, you wanna try all of them. But, will any of these new specific techniques coming out at a different time really help on your task, on your job? Well, these are all just implementation tactics and you wanna put them inside of clear contract. If for your repeated AI task you define an input interface and an output interface You get to play in the internals, you get a lot of agility. Let's make it concrete for AI. So, my first AI program I made when I discovered DSPI was that I had some invoices from my farm and I wanted to extract them to do my taxes. I wanted to extract the tax values from there. Then, another AI program I did is that on my keyboard in my computer I have a little command that reads my keyboard shortcuts, read my clip And will correct the grammar for me. Sometimes, I actually want it to also rewrite for clarity so I have another program that takes text and just rewrites it for clarity, put it back in my keyboard and that's a command and then I can like have a lot of agility and bring it different places. Inside of that, I can change it however I want, the new model comes up and I can change that, it's super easy because my interface is fixed like that. I'll skip that one. But they're not, restrained to very Easy things and small input output. You can be very ambitious with AI programs. So in this examples, you could have your entire inbox and a new email coming in and you wanna compose a new drafted reply. We can do that in DSPIRE with RLM, recursive language models. This is an idea that came from around our community. Or, more like things we probably all do, agentic engineer Your vibe coding, you can give it a spec, a repository, and you get a PR. Those are repeatable tasks. And so, as I have been telling you, when you fix that boundary you can focus on the how on the top and then inside of it, you can have a little chat with just a simple prompt. You can integrate on that prompt. Agents come out, you change it to be an agent. Tools gets invented, you add tools. And then we get into loop engineering, you put that inside of it too. Anything on the outside of it doesn't change. Your integration And and anything else doesn't change. And when you have such a hard boundary, you can also start to automatically optimize. But how can you automatically optimize with just that simple signature? This is not enough. This is not enough to specify your task. And even before ChargePD came out, the creator of DSPI had started to land on this Idea that you need three things to specify your task and if you have this language and this ability to express your task in a programming language, you can start to automatically optimize and delegate away the implementation details. So the first one is what should happen. This is instructions. The signatures that I've been showing you are part of that. Here on the screen, you see the beginning of a real script in DSPI. You set your model at the top, you configure that and it's fully independent of the signatures here where you have natural language instruction to Extract all taxes and, and if it's eligible to output zero. Then you say it, I'm gonna give you an input. It's gonna be a string. I want you to give me an output and it goes It's gonna be a string and a flow. This is natural language expressing my needs. This is very powerful and efficient. If you think about it, if you have a friend over coming to play a board game with you and you give them the instructions and they're ready to play. But if you want to do like alpha go or alpha zero and you tell them You're just gonna learn from example, you're gonna have a long night. And then the second one is what must happen. There are some constraints you have that they have to be listened to, they have to be enforced. The best way to do that is with code. So I want you to go to the third line, a fourth line. You have self dot extract and self dot recheck. You can see we're doing a predict on the extract taxes. And we're doing a chain of thought on the extract taxes. The first one is a vanilla program, the second one makes it do some reasoning. Now, I'm taking them inside in the forward and you can see in the if not print tax, this is a requirement I have that if my first simple vanilla program doesn't extract my taxes, I want you to rerun with more reasoning. I mean, I gotta get my taxes right. And then, another requirement I have is if the value is below zero, True, I wanna show that to a human, I don't want to let you go. This will not change, like even if I have AGI, I would hope it doesn't make mistake, but whatever is in the predictor, if they make these mistakes, I still want these things to be true. So the last one is what good look like. And when I was young, I was on the farm with my dad and I asked him, how do you know that this tree is a maple? And he couldn't tell me. He couldn't give me the instruction on how to know this tree is a maple and he certainly couldn't give me code on how to know this tree is a maple. And so, through time with example, I learned how to know that the tree is a maple, but this is not limited to things like classifying plants. It's also for all of the long tails in your specifications that are things that are more latent. These are sometimes the reason why you would do internship and you would have a mentor and a mentee. You're looking at a lot of Examples and there are long tails of successful behaviors that you have to see and learn. Not that you have all of these, you have expressed fully, you have all these three languages you can put together, you have the specs, the code and the eval and now your goal is fully specified. And so you can start optimizing. You can use things like JEPA on your metrics and on your program and you can start optimizing. At the beginning of the SPAY, the trajectory didn't exist. The models were not good enough. Have to optimize and so we were using code to find few shots examples to make the base models act in the proper way. Then models got better and so we could automatically optimize instruction and in the future, we are starting to be able to be liberated more and more from the implementation details and delegate that away. And at the end, our hope in the Aspire is that you can stick to all of that and then just the news and the implementation details will be automated for you. Isaac will talk to you a lot more about what has been released in the last year, what we're releasing now, and all of the future plans we have. Thank you. Thanks, Max. So we've given you a pretty big abstract overview of specs, code, and evals, but these aren't things that are just restricted to the academic sphere. These are Used in production by some of the biggest enterprises for massive gains. And we see two main benefits when you use DSPine in the enterprise. First, is that your implementation becomes cheaper. When you're flexible to what the implementation is, you can use the bitter lesson to search over different solutions, find something that solves your problem cheaply. And you can use this to scale to data sizes that weren't possible with a more expensive implementation. Shopify, 550 times cheaper. They're able to do that. Because they went from an expensive model to a cheap model, but they could keep the same evals, keep iterating on their business logic inside, and try new things. There's three awesome case studies here, and you should check them out after the talk. They give you a lot of details on how you can do this in your own enterprise. Now, part of the reason why you want to build in the DSPI ecosystem is that we're constantly adding new techniques for you to try. And it's Important to know, none of these techniques we add will definitely solve your problem, because that's your job. What we can do is we can solve sub problems for you that make your implementation easier. For instance, Alex Zhang, a PhD student at MIT, came out with this paper called recursive language models. Recursive language models are a way to solve some kinds of long context programs. And guess what? We can bring this into DSPI for you to try, see if it helps Your long context tasks. Maybe it will, maybe it won't. But the thing is, it's one line, and your signature stays the same. That's what's important here. Everything gets to stay constant, and you get to see if this solves your problem or not. And we've had a number of examples of this just in the last year from people building in and around the DSPI community. We've had RLMs. We've had JEPA, which is an incredible prompt optimizer out of Berkeley. Better together, multi module g r p o. These are incredible research innovations that you get to try in your implementation just by being in the DSPI ecosystem. And we have more coming in DSPI four. I'm excited to talk to you about two of those today, DSPI Flex and Qualitative Learning. DSPI dot Flex is a new kind of module. In DSPI, when we let you optimize things, it started with few shot examples, then it became props, and now that's becoming code. For any function that you want to implement, you can actually learn a harness over time to solve that function. And this is completely custom, and you don't care about the implementation as long as it solves your business problem. Which you've created ways to measure because you've defined the three core parts of specs, code, and evals. The second thing I'm excited to talk about is qualitative learning. One of the hard Hard problems in AI engineering is building evals. And there's a few reasons why this is hard. One is that defining what good looks like is really challenging for any real world problem. The second is that when you define good, oftentimes you have to lose detail. If an email is good or bad, contains a lot less information than if you know what could change in that email in order to improve. And the third is that whenever you create a hill in a data That you're really trying to create a proxy for reality. What if instead we could use reality to inform our evals automatically? What qualitative learning asks is how do we decrease this question? How do we decrease assistance? And it's a research question right now. But what we believe is that models are now good enough to interpret whatever textual feedback is present in the environment, and convert that into evals and a hill that the model can climb. And so as you get more feedback from production, it's traces, it's user actions, it's product analytics, it's asking you it's the model asking you questions about how data should be represented. As you do this, the model can iteratively refine the hill over time and continue climbing it to solve your actual business problem. And DSPI focuses on these kinds of Smile problems. We have a really strong research ecosystem and we collaborate really closely with them. And that's part of the beauty, is that we can see the problems that happen in applied AI engineering. So define them, build a benchmark, and then solve them with techniques. And then we get to democratize the results of that to everyone, because it's open source, open research. Now, one common question is what happens when we have AGI? Well, even when we have an incredibly smart model, the model won't know how to solve your problems. How to do your tasks or have your context. And so this genre of last mile learning is trying to ask, how do we efficiently do this learning? Intelligence is very different from being all knowing. If you were to ask Albert Einstein to help you with your emails, he'd probably ask what's an email. But, if you AGI will know how to Your emails. Nevertheless, it won't know how to actually solve your problem and interact with the people you need to interact with. It won't understand your relationships without learning this context over time. Since 2022, DSPI has been focused on these three core ideas of specs, code, and evals, all defined as a programmatic interface. We've certainly evolved over time, and new techniques are incredible.

## Slides