# The Missing Layers of Conversational AI — Olivier Teboul — session 2026-09-24T15:30:00.000Z → 2026-09-24T15:59:30.000Z _108 transcript lines · 71 slides · source: full recording_ ## Transcript **[00:00:00 · 0]** Was Siri was doing. Like, everyone on his phone can talk to it instead of typing. So underlying those issues are some challenges, and I'm going to pick two of them, to explain, to dig into how how we do and what's the challenge. So if you want to do, voice models, you need to... Several things. **[00:00:28 · 0]** Basically, you want something that's fast, and something you can serve at scale. And so, you have to think about this fast and at scale, this real time at a scale from the ground up. So it means like designing a model that will be able to scale and that will be able to be to serve fast, optimizing the inference, and having a serving infrastructure that will support it. I'm going to talk about the first one and the last one. So how do the... **[00:00:57 · 0]** How does model work? So when you want to design a new model, you don't start from scratch, basically. Right? You start on something that's similar and try to adapt. So what you really need for... **[00:01:13 · 0]** And I'm talking about the speech to speech, system. You need something that will take the speech as it comes And that will be able to take more and more speech as the conversation keep going. And that will be able to, as soon as possible, start speaking in return. Right? And, of course, there is one of the requirements that it's speech in, speech out. **[00:01:38 · 0]** Right? So even if we don't know those models, there is a family of model that is really similar to this and that satisfies two out of those three requirements, which are LLMs. LLMs, you take a text in and a text out. So how do they work? They... **[00:01:54 · 0]** You have a context. At the beginning, it's the prompt, but then it's a context. Those are words. And the LLM is trained to predict what's next. Okay? **[00:02:05 · 0]** So in that situation, Gradium is an AI model company based... What's next? Is it in, off, at, blue, bird? You know, it gives a probability to all of them. So, of course, you know, in seems to be a good candidate. **[00:02:21 · 0]** So if you represent it that way, on the bottom line, this is the context. On the top line, this is the prediction. What's interesting is that, and that's why it's called autoregressive models, the next prediction is going to be part of the context at the next round. So you say, oops, sorry. Gradium is on predicted AI. **[00:02:44 · 0]** AI becomes part of the context. And now it can bring a new one, and another one, and another one. And this way, on the go, you will, predict and generate a full sentence. So can it work on audio? Feels like it's the same, right? **[00:02:58 · 0]** Audio is a sequence. Sequence to sequence. Why could not it work? Well, there is a dimensionality issue here. The problem with audio is that this text, GrainDAM, is an iModel company based in Paris. **[00:03:12 · 0]** It's nine words. So, like, eight words for the context one you're gonna predict. It's three seconds when it's spoken out loud. But three seconds of audio, if you represent it on a computer, on a normal quality, it will be like 24,000 samples per second. Those those eight words becomes seven... **[00:03:32 · 0]** 72,000 sample, values. So it's much longer of a sequence. So we have 10,000 times more, values to predict. And since the transformer architecture, which what's the LLM is relying upon, is relying on self attention which is quadratic. Then your, attention metrics is a 100,000,000 times bigger. **[00:04:00 · 0]** So it's not tractable. So the first trick is to turn the audio into tokens. And that's the neural codec. And actually, it turns out you can do that. You can train a model to compress the audio in tokens. **[00:04:16 · 0]** From 24 kilohertz to 12 hertz. 12.5. So divide by 2,000. Then you have a tractable number of tokens. That's a sequence and you can start predicting because at the end of the story, we'll decode them into... **[00:04:31 · 0]** Back to audio. So if you wanted to do that to train a dialogue system, you could say: me and the... I'm the LLM. Okay? I'm the gray one. **[00:04:43 · 0]** So we have a sequence of me talking, me talking, me talking, you talking, you talking, me talking, blah blah blah. That's a sequence. And we're gonna predict the next audio tokens. Right? But this... **[00:04:55 · 0]** Sorry, sorry, sorry, sorry... This doesn't really work like this. Because in a conversation, it's not like I talk or you talk. We might be talking at the same time. Otherwise, it's just like walkie talkie conversation, which is lame. **[00:05:09 · 0]** So in practice, you have to invent something new that we don't have in LLMs, is multi stream hierarchical transformers. So, like, now you don't have a single line of time. For the context, you have two. You have me speaking and you speaking. And those two things, I will attempt to to be able to predict what's next. **[00:05:27 · 0]** And so those models, they can listen and talk at the same time. Give you an example. Sorry. Of course. **[00:05:53 · 1]** So, the planet is Sirius 22. Can you plot a trajectory course to it, please? **[00:05:58 · 2]** Yes, sir. **[00:06:00 · 1]** Okay. How long is it gonna take us to get there? **[00:06:02 · 3]** I've mapped it out. It's approximately five months to get there. **[00:06:05 · 1]** Okay. That's that's not too bad. Do you think we have all we need onboard the ship to start the mission? **[00:06:10 · 3]** Yes, sir. We have everything we need Okay. Good. Today. **[00:06:13 · 0]** Okay. I'm gonna skip the second one. So you see, like, it's very natural that the model is trained to listen and talk at the same time, so there is no latency whatsoever. And that was Alex, our chief scientific officer, in 2024. So now you're... **[00:06:28 · 0]** There is something I didn't tell you. It's that in 2024, we had this technology that's QTie, but we're not selling it. So what's the missing layer here? Well, there is something here is that this one model is just one model, and it's capable of listening, speaking, and, you know, processing language. That's a lot of things for a very small model. **[00:06:54 · 0]** So when you talk to it, at some point you will think, okay, it's not as good as, you know, the limitation is even worse. In that this model is kind of air gapped. It doesn't talk to the external world. So if it doesn't talk to the world, if it cannot act on, you know, booking an appointment or something, In production, it's just a gadget. So the big challenge for speech to speech today is to be able to do seamless integration with tool calling. **[00:07:21 · 0]** And if you do that, then you get the same experience while being able to do what a voice agent should do. Okay. So that was about how we build a model, roughly, not all the recipe. And now let's say you have the model and you want to scale it and serve it to the world. How do you do that? **[00:07:44 · 0]** So here we got the latency problem. So you have one person with his computer. I'm going to take the text to speech example here. Sending some text to our server, and we have to answer some audio back. And if you want to plug that into a voice agent, it has to be fast. **[00:08:05 · 0]** It has to be real time. And by real time, we mean less than three hundred milliseconds. So that's the latency budget that you have for the whole thing. If you don't, then it's gonna be like a weird conversation where you wait for a while before getting an answer, and you're gonna be bored. So it has to be fast. **[00:08:24 · 0]** And during that trip, you have actually three layers that you have to go through. One is the transport layer. Right? Network layer. The information has to travel from point a to point b. **[00:08:39 · 0]** Then we have the product layer. Because, like, someone sends me a request. I should check. Is is that legitimate? Does this person, does this person have the permission, the credits? **[00:08:52 · 0]** He wants this voice. Do we have this voice? Where is it? And so on and so forth. So you have to enhance the request or deny the request. **[00:09:00 · 0]** And then there is the compute. So how do we do that fast? So first problem you have to solve is the latency problem. So let's say I put my server somewhere in Europe and my client is somewhere in The US. Well, just by going round trip through the Atlantic Ocean, I kind of... **[00:09:21 · 0]** I'm wasting a hundred milliseconds, one third of my budget, for just transporting the information. So how do I fix it? Okay, easy. I'm gonna put servers everywhere, right? It's more expensive at first, but I'm gonna reduce the trip from the client to the server. **[00:09:41 · 0]** But now I got new issues, right? Because I had the system with no replication and no distribution. I have distributed systems where the information has to be synchronized. Of course, if you want to go fast, you put caches. Right? **[00:09:56 · 0]** So I put... I have a database. I have cache of my database. It's already an issue per se for, you know, all the cache invalidation. But now you have many caches that has all to be synchronized altogether. **[00:10:06 · 0]** Otherwise, you create an API key and you're going to California. It doesn't know about the API key. That's weird. Okay? So you're trading latency for complexity. **[00:10:18 · 0]** Engineering complexity. We can handle it, but still it's it's important to be aware of it. Then the second thing, which I think is very very new. You know, when we started a year ago, we were thinking, there is no way if I need a new GPU to handle my request. There is no way I can ask my cloud provider for a GPU. **[00:10:41 · 0]** He give me your GPU or she give me a GPU. Their GPU is ready. Then I pull the weights. I pull the Docker image. I warm up the GPU and this gonna happen in three hundred milliseconds. **[00:10:53 · 0]** Impossible. Right? So like if you try, you you can maybe make it minutes. Right? Today what's happening... **[00:11:01 · 0]** So we we never put that as a... As an option. You have to predict and to be ready for the load. But now what's even worse is that the GPU scarcity is so important that you may not be able to have it even, like, in next day. So maybe you want one more GPU but you don't have this GPU because there is no GPU available in your region. **[00:11:24 · 0]** Because one of the cloud provider are saying, sorry. There is... Everyone wants a GPU and we don't have it. So how do you handle that? So again, you're gonna trade, you're gonna mitigate this risk with more complexity. **[00:11:38 · 0]** And there are two things you can do. The first thing is I'm going to go to different providers. There is more the odds is... The odds to find the GPU is better. Or I'm going to try to, have my model run on different architectures, on different models, on different chip providers. **[00:11:57 · 0]** Right? Not only Nvidia or not only h one hundred or stuff like that. So that increases the odds of finding a GPU. So, again, to mitigate that risk, I'm increasing the complexity of my system. And how do you handle, like, burst? **[00:12:12 · 0]** Right? Because you can, sometimes, everybody wants at the same time to be served. Right? And there are different strategies you can do. And we take two options. **[00:12:24 · 0]** First one is to say, now let's take advantage of all those clusters and reroute the traffic from one to the other, the closest one. Or you can also do... Change the batch size of your model. Maybe you can trade a little bit of latency with a bigger batch size. Right? **[00:12:42 · 0]** For the time where you hit, you know, the wave, you increase the batch size and then you decrease it. In practice, if you put all that one after the other of good model, fast inference, fastly served. That's what you get. So we did, we're building this week a new beta model for text to speech. And we did that race where we click somehow, not at the same time. **[00:13:07 · 0]** What we measure is the time for, how much it takes to get the audio back for us and some providers. And this is what we we got. **[00:13:21 · 4]** Hi. I'm Gradium's newest model, and I'm the fastest text to speech we've ever built. Faster first audio, more natural conversations, no waiting, and guaranteed speed. Try it today. **[00:13:34 · 0]** Yeah. So we're happy that we're, you know, pushing the boundaries of of latency. You know, the best models were about, like, hundred fifty milliseconds, and we made it, like, three times faster. So at some point, there is no there is no limit. Right? **[00:13:48 · 0]** We... It's not... Never gonna be, you know, one zero milliseconds. But we're happy that we're able to put all that together, to make it happen. Now to close this talk, I'm going to take talk about... **[00:14:04 · 0]** Can we go further actually? I'm I told you we cannot go zero milliseconds and that's true. But can we go zero milliseconds of travel? Can we go zero milliseconds of product and still have something fast that would scale even more? And that's the idea of moving the compute from the server to the client directly. **[00:14:25 · 0]** And for this, you have to have very tiny models that can run on your device, on your phone, on your brother, on a Raspberry driver. So we also released, like, Gradient Fonon, which is the smallest. It's not exactly the smallest. One of the smallest, but with a very low number of parameters. A 100,000,000 parameter only, and a very good accuracy. **[00:14:52 · 0]** And this you can also try. Feel free to reach out if you wanna try those models. I'm gonna give you a little bit of examples of what it sounds. **[00:15:01 · 3]** Only 100,000,000 parameters, and I **[00:15:04 · 5]** can speak in any voice you give me. **[00:15:07 · 6]** And here's an example with a brand new voice, cloned from a ten second sample. **[00:15:13 · 0]** Okay. We can skip... **[00:15:14 · 3]** This voice didn't exist a minute ago. I generated it from a single ten second recording. **[00:15:21 · 6]** Same model, completely different voice. **[00:15:24 · 0]** You got another... **[00:15:24 · 6]** Cloned on the fly from a short clip. **[00:15:28 · 0]** Alright. So that's that's pretty much all I wanted to, tell you about. What's below the hood in in the voice, AI? How we build conversational agents? How we train those model, how we serve those models, and how if we, you know, put together nice research and nice engineering, we can really, push the boundaries of of conversational AI. **[00:15:52 · 0]** Thank you. **[00:15:59 · 7]** Thank you so much, Olivier. Let's give it up for Olivier for... And Gradium. Nice. **[00:16:05 · 0]** Oh, **[00:16:07 · 7]** alright. Okay. So if you follow the QR code that was, that was shared with you guys, then you can have, credits, right, to to try out Gradium. Alright. Thank you. **[00:16:22 · 7]** Okay. So, we still have a few more things for you guys, so don't don't go... Don't leave just yet. Actually, I've been working in AI for, quite some time now. I can't say that it was a long time, but it feels like a long time. **[00:16:36 · 7]** And one of the questions that I get the most is, what do you think is gonna happen to the next generation? And it's a tough question. Right? Because things are moving so fast, and it's already quite hard for us as adults to understand the impact of all this technological change. But luckily for us, there are people that are doing something about it. **[00:16:56 · 7]** And, I I would like to welcome Cassandra Chin, who's gonna tell you more about it. So please, let's give it up for Cassandra. **[00:17:11 · 2]** Hi. So my name is Cassandra Chin, and I am one of the kids workshop instructors for AI Engineer Paris. This weekend, we ran a kids event where we educated kids and helped teach them to learn about technology to prepare them for this AI world. But before I get too much into it, I wanna talk a little bit about myself and why I'm so passionate about teaching kids technology. I actually started teaching kids when I was a kid myself. **[00:17:42 · 2]** I was 13 years old at the time, and I would teach kids workshops over the weekend, Raspberry Pi workshops where kids would get to tinker with different technologies. But even before I started teaching, in my household, there was lots of technology. We had three d printers, a lot of different technology. I remember building my own computers. I really believe in inspiring kids to enjoy technology rather than pushing them. **[00:18:12 · 2]** Because if you show kids that technology is this very fun thing, not just dad or mom at the computer. Like, show the kids what technology can actually do. That's what we really tried to do in these kids' workshops. And for the kids' workshops this weekend, I taught a workshop teaching kids how to code with AI. The kids used Mistral Live to build their own games, and I was really surprised at what the kids could do. **[00:18:42 · 2]** Because we have these preconceived notions about how to code, what you can or can't do. But the kids, they just have ideas. Some ideas they have are, like, building an RPG game, a platformer, a banana game, a lot of creative ideas from kids, but they just prompt and the AI could build it for them. And with, we had an organization called AI Tinkers. They brought in a lot of underprivileged kids for, to the workshop, and these kids, they they could learn really well. **[00:19:21 · 2]** Like, it didn't matter what background they have, and I think it was really great that they also brought in a lot of girls. Another issue with technology is there's a lot of gender biases. As kids get older and older, they start thinking that technology is a guy's thing because they see more of the industry. But if you catch kids when they're very young, they don't have this idea that technology is a guys' thing, and a lot of girls can be inspired to join technology like I have. So I think these kids workshops are really great for kids. **[00:19:55 · 2]** The workshop which I taught, we had the kids build the games, and we actually connected it to a GitHub repo, which had a lot of our assets and, agents dot MD file. So this created a more kid safe way with AI because of huge concern is how is it safe to let kids use AI. But with this agents dot MD file, we were able to instruct the AI to act more friendly, use less tech jargon, and be more helpful, run commands rather than asking you to run commands, a lot of instructions which make the workshops more kids' friendlies. We have this video which we produced about the kids' workshop. So if you would please watch it. **[00:20:41 · 2]** I think it's really nice. **[00:20:46 · 8]** We're here at AI Engineer Kids' Day in Paris teaching the local kids about AI technology, about models, about neural networks. This is a great way to expose kids to AI in a fun way, in an approachable way, and something we love doing at Neo4j because we're helping train the next generation of AI engineers. **[00:21:12 · 9]** There's a whole mix of kids here, from a bunch of different backgrounds. Some of them are already expert coders at eight years old, and some of them are in their teams and they need a little bit extra help. But that's kind of what excites me is we're putting them all in one room. There's peer to peer learning, they're teaching each other, and they're helping each other. So that when they grow up and they start working in the industry, they'll have a handle on their own lives, their own careers, their own ideas. **[00:21:41 · 2]** For my workshop, the kids use Mistral. Lib and it is a coding agent which lets them create their own games. It's really interesting what kind of games the kids build. Some of them choose to build an RPG game, some do platformers. Another kid did a balloon popping game. **[00:22:01 · 2]** And what's interesting is the... There's no language barrier. Even so the kids mainly speak French, they can type French to the agents and it fully understands French and still codes the games for them. So I think this was a really good experience for the kids because they got to build their own unique games. **[00:22:27 · 7]** So my organization AI Tinkers organizes a lot of events and having an event that will bring together the kids and their family members. So our usual community members that could be the parent or the big brother or big sister. That sounded really exciting so that they can build together, and I believe that's part of community building. That's the reason why we were really excited about doing this. **[00:23:14 · 2]** We are running these kids workshops as a part of the AI Engineer Conference Series. So we will be running the next workshop at aiengineernet... New York, and also AI Engineer Code in San Francisco. And next year we'll be running it at London. So if you're at those events, please bring your kids. **[00:23:35 · 2]** We'll welcome your kids and teach them. **[00:23:43 · 7]** Thank you. Yeah. Thank you, Cassandra. Let's give it up one more time. That was awesome. **[00:23:47 · 7]** Yeah. Thank you, Cassandra. Thank you, Neo four j, for doing, yeah, such a great job. I'm a father myself, and I can relate. This is super important work. **[00:23:58 · 7]** Yeah. So thank you one one more time. And, let's hear it from you guys as well. Thank you. You you made it. **[00:24:04 · 7]** Hey. You made it all the way here. Right? I know somebody somebody told me earlier is like, hey. You clap too much. **[00:24:11 · 7]** My hands are hurting. I'm like, hey, dude. Like, we we gotta find out... Oh, you're there. You recognize yourself. **[00:24:16 · 7]** Nice. Yeah. But thank you so much for being with us, and I would like to, thank everybody that helped us organize this event. Right? I would like to, sorry. **[00:24:30 · 7]** I would like to thank our sponsors. Mistral first for organizing the event, and then NVIDIA is our platinum sponsor, but also all our gold and bronze and and silver sponsors. So let's give it up for them, please. And also, I think you should be proud of yourselves. So I I heard that many of you have traveled all the way here from different places. **[00:24:56 · 7]** We heard somebody from French Polynesia and then Australia to Lviv and different parts of Europe as well. I think this is so cool that people come from different parts of the world to come and and and be with us here, and, we really appreciate it. So, yeah. So let's let's hear it from you. Let's give it up for you. **[00:25:14 · 7]** I think you did a good job. Okay. So before we close it out, first of all, I hope I see you next year. But we have one more thing from Patrick and our sponsor, Backblaze. So, yeah, let's hear it for Patrick. **[00:25:31 · 7]** Alright. **[00:25:33 · 10]** Alright, guys. The first thing I'm gonna promise you is this will be exactly two minutes, and let's give it up for how much he claps, because you gotta clap one more time. Can you **[00:25:43 · 0]** do it? Alright. **[00:25:46 · 10]** So, I'm Patrick from Backblaze. Great to meet you all. I am buying you drinks tonight. So I am contractually obligated to explain to you what is Backblaze. I'm gonna do it as simply as I can. **[00:25:58 · 10]** I'll give you one sentence to remember. AI runs on GPUs. GPUs need data. Backblaze is the best capacity storage tier for AI data. So if you remember anything, remember that. **[00:26:11 · 10]** Two examples. Descartes, the world model builder. When they were getting their start, they, Dean called me up and he's like, hey. I've got 15 petabytes and I'm moving it in days, and I haven't broken you guys yet. You're awesome. **[00:26:25 · 10]** And I was like, awesome. Who are you? What are you doing? I have no idea. But they chose Backblaze because we don't charge egress and we have really incredible throughput. **[00:26:35 · 10]** So they were able to race different GPU clusters and create incredibly efficient AI training runs. Another customer, CoreWeave. We are their object storage tier. They chose us because they ran a very exhausted POC. And what they found was that, from exabytes upward, our economics and our performance did not change. **[00:26:58 · 10]** And so they selected us because they believe we could scale with them forever. So whether you're working with terabytes or exabytes, we are a great solution, to ensure that your data is someplace where you control it. You can use it how you want to. You can use it with who you want to. And, as an s three compatible storage layer that is a quarter of the cost of s three and has free egress, you're gonna pay a lot less to do it. **[00:27:24 · 10]** Like I said, our mission, which we execute for lots of Gen AI companies and NeoClouds and data scrapers and physical AI companies, is use your data however you want, wherever you want, with whoever you want. We are only gonna focus on storage, and we're gonna make sure every byte is there as fast as you need it. And our primary goal is that you spend as little money as possible on storage because after listening to all the talks here for the last couple of days, what you guys are doing is so interesting, and storage should be the last of your concerns. So definitely check us out. I will be upstairs. **[00:28:04 · 10]** We have a signature, cocktail, Lebac Blaze. Very original. Try one. Talk to me. And if you are early stage, we also have a startup program where we offer up to a $100,000 in storage credits, which if you can do the math, we're a quarter of the cost of s three is gonna last, a good amount of time depending on how fast you're growing. **[00:28:26 · 10]** So thank you, everyone. Thank you again to AI Engineer and Mistral and all of the sponsors, and especially thanks to you guys. I flew here from Minnesota two nights ago. I'm flying back tomorrow morning. I'm exhausted. **[00:28:38 · 10]** I have to go take care of my kids. I know how hard this is, but I think it was totally worth it. So thank you, and we'll see you upstairs. ## Slides ### 00:00:00–00:00:20 [A dark, blurry background image featuring glowing orange lights and a large, dark, mechanical object on the right, possibly a robot or part of a vehicle.] ### 00:00:20–00:00:40 ## Underlying scaling challenges - A model **designed** for scale - Fast inference on **large batches** - A scalable **real-time** serving infrastructure ### 00:01:00–00:01:20 ## Grounding based ### 00:01:20–00:01:40 # Background on text models LLMs are **probabilistic** models predicting the next **word** given context [Diagram showing a flow: a text input "Gradiam is an AI model company based..." feeds into a "Text LLM", which then outputs a list of words ("in", "of", "on", "by", "at", ...) each with an associated horizontal bar representing its probability.] ### 00:02:20–00:02:40 # Background on text models LLMs are **probabilistic** models predicting the next **word** given context [Diagram showing an input box "Gradium is an AI model company based..." feeding into a "Text LLM" box, which then outputs a list of words ("in", "of", "on", "by", "at", ...) with an accompanying bar chart indicating their probabilities.] ### 00:02:40–00:03:00 # LLM [Diagram showing vertical bars and an arrow, possibly representing a process or data flow] ### 00:03:00–00:03:20 # Background on text models Can we just replace **text** with **audio**? [Diagram showing audio input, an arrow to a "Text Audio LLM" block, an arrow from the block, and audio output] ### 00:03:20–00:03:40 # Background on text models > "Gradiam is an AI model company based in Paris." Each prediction requires computing the Transformer (7B params!) **Text** 8 words **Audio** 3 seconds = 72,000 values at 24kHz **10,000 times more tokens, 100M times more costly self-attention!** ### 00:03:40–00:04:00 # 4kHz ## ...es more c... self-at... ### 00:04:00–00:04:20 # Background on text models > "Gradiam is an AI model company based in Paris." - Each prediction requires computing the Transformer (7B params!) - **Text** 8 words - **Audio** 3 seconds = 72,000 values at 24kHz - 10,000 times more tokens, 100M times more co... self-attention! ### 00:04:20–00:04:40 # Background on text models Our **neural codec** compresses audio from 24kHz to **12.5Hz** [Diagram showing a waveform icon labeled "72,000 samples" flowing into an "Encoder" block, which then outputs three green squares labeled "37 tokens"] ### 00:04:40–00:05:00 # Training an audio LLM for dialogue - `[Grey rectangle]` = LLM - `[Green rectangle]` = USER [Diagram showing a large central box labeled "Audio LLM". Above the box are three stacked grey rectangles. Below the box is a sequence of stacked rectangles, alternating between grey and green, representing turns in a dialogue: five grey, then two green, then one grey, then one green, then one grey.] ### 00:05:00–00:05:20 # Background on text models Does not model overlap of real conversations! - = LLM - = USER [Diagram showing a large green-bordered rectangle labeled "Audio LLM", with two stacked green rectangles (USER) above it, and a sequence of stacked grey (LLM) and green (USER) rectangles below it. An icon of an old mobile phone is in the top right.] ### 00:05:20–00:05:40 [Diagram showing a bright green L-shaped bracket in the upper left, with several layered rounded rectangular shapes in shades of grey and green below it.] ### 00:05:40–00:06:00 # /kyutai [Screenshot of a dark-themed application interface showing audio controls, server audio statistics, a dialogue transcript, and an embedded video feed of a speaker.] ### 00:06:00–00:06:20 ### Interactive Application Demo [Screenshot of a web application displaying server audio statistics, a large circular audio indicator, a series of smaller rectangular indicators, and a chat interface with a conversation.] ### 00:07:00–00:07:20 # Current speech-to-speech limitations - Moshi is one model, that will never be as smart as big LLM - No tool calling, no RAG The challenge of speech to speech is the seamless integration of tool calls into the conversation. ### 00:07:20–00:07:40 ### Current speech-to-speech limitations - Moshi is one model, that will never be as smart as big LLM - No tool calling, no RAG The challenge of speech to speech is the seamless integration of tool calls into the conversation. ### 00:07:40–00:08:00 # A scalable real-time serving infrastructure [Abstract ring-like shape composed of many small green and blue dots] ### 00:08:00–00:08:20 # The latency problem We serve realtime models powering voice agents. The first bytes should be received in **less than 300ms**. How to ensure that: - everyone is served in milliseconds - we can handle bursts hot path < 300ms 3 layers: transport, product, compute auth, authz, resolve voice, GPU routing, metering, etc. [Diagram showing a laptop sending "text" to a stack of servers, which then sends "audio" back to the laptop, labeled as a "hot path < 300ms". Below this, "3 layers: transport, product, compute" points to a list of backend processes.] ### 00:08:40–00:09:00 # The latency problem hot path < 300ms We serve realtime models powering voice agents. The first bytes should be received in **less than 300ms**. How to ensure that: - everyone is served in milliseconds - we can handle bursts 3 layers: transport, product, compute auth, authz, resolve voice, GPU routing, metering, etc. [Diagram showing a laptop sending "text" to a server stack and receiving "audio" back, illustrating a "hot path" with latency less than 300ms] ### 00:09:20–00:09:40 # The latency problem: the transport layer The client maybe far away from the server. How to ensure that total round-trip stay below the 300ms threshold ? [Diagram of the Earth showing a client in North America and a server in Europe, connected by a line labeled "100ms"] ### 00:10:00–00:10:20 # World-wide presence The server has to be located somewhat near the clients to reduce transport time. New issues arising from the mesh architecture: - How to synchronize the information (voices, usages, etc.) ? [Illustration of a globe showing server and client locations connected by lines, representing a worldwide network architecture.] ### 00:10:20–00:10:40 ## World-wide presence The server has to be located somewhat near the clients to reduce transport time. New issues arising from the mesh architecture: - How to synchronize the information (voices, usages, etc.) ? [Illustration of a green globe with points marked as "server" and "client" connected by lines, representing a worldwide distributed network.] ### 00:10:40–00:11:00 # Reference support ### 00:11:00–00:11:20 # World-wide presence in a world of GPU scarcity Scaling up in milliseconds is not possible. This has always ruled out real-time autoscaling. Today, scaling up on a public cloud can take hours, due to GPUs availabilities. How to cope with it? - Multi-cloud providers - Multi-architecture inference support [Map of the world showing server and client locations connected by lines] ### 00:11:20–00:11:40 ## World-wide presence in a world of GPU scarcity Scaling up in milliseconds is not possible. This has always ruled out real-time autoscaling. Today, scaling up on a public cloud can take hours, due to GPUs availabilities. How to cope with it? - Multi-cloud providers - Multi-architecture inference support [Illustration of a globe with points labeled "server" and "client" connected by lines, representing a distributed network] ### 00:11:40–00:12:00 # World-wide presence in a world of GPU scarcity Scaling up in milliseconds is not possible. This has always ruled out real-time autoscaling. Today, scaling up on a public cloud can take hours, due to GPUs availabilities. ### How to cope with it? - Multi-cloud providers - Multi-architecture inference support [World map with points labeled "server" and "client" connected by lines, indicating a distributed network] ### 00:12:00–00:12:20 # World-wide presence in a world of GPU scarcity Scaling up in milliseconds is not possible. This has always ruled out real-time autoscaling. Today, scaling up on a public cloud can take hours, due to GPUs availabilities. How to cope with it? - Multi-cloud providers - Multi-architecture inference support [Map of the world showing distributed server and client locations] ### 00:12:20–00:12:40 # How to handle burst ? - Real-time peer-forwarding cross region. - Batching strategies Watch out for maintaining data residency! [World map showing a client in North America connected to multiple servers across North America, Europe, and Asia, with servers also connected to each other.] ### 00:13:00–00:13:20 # The TTS race: new beta is out! [YouTube play button icon] ### 00:13:20–00:13:40 # The TTS race: new beta is out! ### 00:14:20–00:14:40 [Abstract pattern of green, blue, and purple dots forming a curved band] ### 00:14:40–00:15:00 # Gradiium Phonon: Smaller, better ## TTS models: size vs WER - Voice cloning is not used in this benchmark ### Device - iOS, Android, Handheld devices ... ### Inference Mode - CPU-only, faster than real-time ### 00:15:20–00:15:40 # Gradium Phonon: Smaller, better [Abstract visualization of a glowing green and yellow circular object with radiating lines, resembling a sound wave or phonon. Three red play buttons are present on the slide.] ### 00:15:40–00:16:00 # Try us for free ## Join us [gradium.ai](https://gradium.ai/) [github.com/gradium-ai](https://github.com/ ### 00:16:00–00:16:20 - AI Engineer Paris - Backblaze - CLOUDFLARE - Cockroach Labs - CodeRabbit - cominty - Daytona - descope - FACT ### 00:16:20–00:16:40 - AI Engineer PARIS - Notion - SIERRA - Mistral - NVIDIA - langfuse by ClickHouse - descope ### 00:16:40–00:17:00 # Partners and Sponsors - NVIDIA - AI Engineer PARIS - Cockroach Labs - Mistral - FACTORY - SENTRY - Sonar - CLOUDF ### 00:17:00–00:17:20 - AI Engineer Paris - Notion - Silk - Mistral - AI Engineer Paris - NVIDIA - langfuse by ClickHouse - NVIDIA - comi... - ### 00:17:20–00:17:40 - Mistral - AI Engineer PARIS - langfuse - Sonar - NVIDIA - cominty [Logos of Mistral, AI Engineer, Langfuse, Sonar, ### 00:17:40–00:18:00 - Mistral - AI Engineer PARIS - langfuse by ClickHouse - Sonar - NVIDIA - pminty - de [NVIDIA logo] ### 00:18:00–00:18:20 - Mistral - AI Engineer PARIS - langfuse by ClickHouse - Sonar - NVIDIA - ninety [Logos and names of various companies/projects, including the NVIDIA eye logo] ### 00:18:20–00:18:40 - stral - AI Engineer - NVIDIA - gfuse by ClickHouse - AI Engineer - NVIDIA - desc ### 00:18:40–00:19:00 - Mistral - AI Engineer Paris - NVIDIA - Cloudflare - langfuse by ClickHouse - Sentry - Modal - Qdrant - Descope [Grid of company logos, likely sponsors or partners] ### 00:19:00–00:19:20 - AI Engineer - NVIDIA - Infuse by ClickHouse - AI Engineer Paris - Media - Descale ### 00:19:20–00:19:40 AI Engineer NVIDIA ClickHouse AI Engineer PARIS desc( ### 00:19:40–00:20:00 - Mistral - AI Engineer PARIS - NVIDIA - CLOUDFLARE - langfuse by ClickHouse - Sentry - AI Engineer PARIS - Modal - NVIDIA - Qdrant - descOpe - AI Engineer PARIS [Grid of company logos] ### 00:20:00–00:20:20 - Mistral - AI Engineer PARIS - Langfuse by ClickHouse - Sonar - NVIDIA - inty - de [NVIDIA logo, a stylized eye ### 00:20:20–00:20:40 - Mistral - AI Engineer PARIS - langfuse by ClickHouse - Sonar - NVIDIA - minty - de [NVIDIA logo] ### 00:20:40–00:21:00 - Mistral - AI Engineer PARIS - Singfuse by ClickHouse - NVIDIA - AI E... - ...nar - ...nty - ...des ### 00:22:40–00:23:00 ### Screenshot of a coding environment [Screenshot of a laptop screen displaying a dark-themed code editor or console window, partially obscuring a lighter background application with UI elements and a cartoon illustration.] ### 00:23:20–00:23:40 - Mistral - AI Engineer PARIS - langfuse by ClickHouse - Sonar - NVIDIA - cominty [Logos for Mistral, AI Engineer, langfuse, Sonar, NVIDIA, and cominty are displayed] ### 00:23:40–00:24:00 - Mistral - AI Engineer PARIS - LangChain - Sonar - NVIDIA - Cominty [Logos of Mistral, AI Engineer, LangChain, Sonar ### 00:24:00–00:24:20 PARIS Mistral langfuse by ClickHouse NVIDIA [Logos of Mistral, langfuse, and NVIDIA] ### 00:24:20–00:24:40 - AI Engineer PARIS - Mistral - Notion - langfuse by Click - NVIDIA [Grid of company logos and names] ### 00:24:40–00:25:00 # AI Engineer Paris Partners - NVIDIA - AI Engineer Paris - Cockroach Labs - Mistral - FACTORY - AI Engineer Paris - SENTRY - ### 00:25:00–00:25:20 - Blackblaze - Notion - AI Engineer PARIS - NVIDIA - CLOUDFLARE - Sonar - Mistral - langfuse by ClickHouse - com ### 00:25:20–00:25:40 # AI Engineer Paris Partners / Sponsors - NVIDIA - Cockroach Labs - Mistral - FACTORY - SENTRY - Sonar - CLOUDFLARE - Daytona - langfuse by Cockroach - Modal - neo4j - Qdrant - CodeRabbit - cominty - descope - Latitude - FIMO - MAKORA - stripe - ZenML labs - Backblaze - Notion - SIERRA [Grid of company logos for the AI Engineer Paris event] ### 00:25:40–00:26:00 # AI Engineer PARIS - NVIDIA - Son... - comint - scope - CLO - se - al [Logos and names of various companies and the ### 00:26:00–00:26:20 # Visible Sponsors/Partners - AI Engineer PARIS - NVIDIA - CLOUD - Sonar - cominty [Logos for AI Engineer, NVIDIA, CLOUD, Sonar, cominty, and other partially visible companies] ### 00:26:20–00:26:40 # Sponsors / Partners - AI Engineer PARIS - NVIDIA - ClickHouse - Sonatype - Engineer PARIS - Cominty - Scope [Logos of various companies including NVIDIA ### 00:26:40–00:27:00 - AI Engineer PARIS - NVIDIA - CLOUD - Son... - AI Engineer PARIS - comin'... - escope - AI E... [Logo for AI ### 00:27:00–00:27:20 - AI Engineer PARIS - NVIDIA - Sona... - ...Engineer PARIS - comint... - ...escope - CLO... - AI... [Multiple company logos displayed on a dark background] ### 00:27:20–00:27:40 # AI Engineer PARIS - NVIDIA - Sonar - cominty ### 00:27:40–00:28:00 - AI Engineer PARIS - NVIDIA - Sonar - cominty - AI Engineer S - c()pe [Logos for AI Engineer PARIS, NVIDIA, Sonar, cominty, AI Engineer S, and c()pe] ### 00:28:00–00:28:20 - AI Engineer PARIS - NVIDIA - SonarSource - comint - scope ### 00:28:20–00:28:40 # AI Engineer PARIS - NVIDIA - Sonar - cominty - c(o)pe ### 00:28:40–00:29:00 - AI Engineer PARIS - NVIDIA - Sonar - cominty ### 00:29:00–00:29:20 # AI Engineer PARIS ### 00:29:20–00:29:40 # AI Engineer Paris ### Organized by - Mistral ### Platinum - NVIDIA ### Gold - Cockroach Labs - Factory - Sentry - Sonar ### Silver - Cloudflare - Daytona - Langfuse by ClickHouse - Modal - Neo4j - Qdrant ### Bronze - CodeRabbit - Cominty - Descope - Latitude - Makora - Fimo - Stripe - ZenMLabs ### Supporting partners - Backblaze - Notion - Sierra [Logos of sponsoring companies grouped by sponsorship tier]