# The Efficiency of Gemma 4 — Maarten Grootendorst — session 2026-09-24T15:00:00.000Z → 2026-09-24T15:30:00.000Z _76 transcript lines · 71 slides · source: full recording_ ## Transcript **[00:00:01 · 0]** And so that explains that the E actually stands for effective because that's per layer embedding lookup tables billions of parameters, right? But they're not being used during inference. We can separate in a way part of the knowledge from actual model capabilities. And this idea of separating them is becoming more and more pronounced in the field of open models. So we have knowledge. **[00:00:25 · 0]** You can do knowledge efficiently. What about size, for instance? You can create a very small model, but you still want to have a lot of capabilities. So what you can do is you can do quantization aware training. You reduce the precision to something smaller while trying to maintain the capabilities as much as you possibly can. **[00:00:44 · 0]** And that's a very nice way to approach this, but we can actually take it a step further with mobile quants. And that's a very specific schema that was developed for these two smaller models because when you have smaller models and you're going to quantize them to even smaller size, you have to make sure that you don't lose too much performance. So a very specific schema and blocks a build like this. We have the token embedding layer. You can actually reduce that all the way to two bits and retain a lot of the capabilities of the model still. **[00:01:17 · 0]** The reason I will come into a little bit later. For the global attention, four bits. Four bits is what we generally use. It's a nice balance between the size of the model and capabilities. But for local attention, which zooms in on the thing that we want to know more it helps to have a little bit of a higher precision, especially with the kv cache. **[00:01:41 · 0]** Feed forward network is four bits, but then the per layer embeddings that we just talked about, those are two bits because they kind of compensate for the lower position of the token embeddings as well. And so when you combine all of this, you have a nice schema for a specific model. And what you see is that making sure that it's efficient also means doing a lot of customizations to these models, especially at sizes like 2,000,000,000, 4,000,000,000 parameters. And with all of that, you have knowledge, you have efficiency in size. What about speeds? **[00:02:19 · 0]** Well, with speeds, we use something that's becoming more and more popular fortunately these days, speculative decoding with multi token prediction. That works by using a rather small model to kind of suggest tokens that the bigger model, the model you want to actually run, can use because that allows the bigger model to then only have to validate all of the tokens that it sees without having to process all of those tokens themselves at once. So we have a target model, right? It's a big one. It can be whatever architecture that you have. **[00:02:54 · 0]** And a draft model, a smaller one. And with Gemma four, what we aim to do is do some kv cache sharing. Why? Because that simplifies the process for the smaller model, the candidate model or the draft model, much, much more. Because what happens is that the target model takes in inquiry, does a single round of processing, one single pass, generates its kv cache, and then shares it with the smaller model. **[00:03:20 · 0]** And the smaller model can then use the output of that and continue on processing and generating tokens. It will take the inputs, generate one, two, three, perhaps eight tokens for the larger model then to validate. And so the larger model only has to look at all of these tokens at once and decide, Okay, I like these tokens, but I don't like so much these tokens. So I'm only going to accept these first two, for instance. But it's still a single pass, right? **[00:03:49 · 0]** So all it needs to do is then add another token because it can. It did the computation for the entire sequence anyway. And then it generates three tokens almost at the cost of a single one, and that speeds things up tremendously. Now obviously nothing is a free lunch, right? So there are pros and cons to a method like this: You add a model. **[00:04:14 · 0]** It's a very small model, but you add it anyway. It also differs greatly on the use case, the speed up that you get from something like this. The thing is some tokens are super easy to predict, especially in structured tasks like code, for instance. But in creative tasks, it's very hard for the smaller model to really figure out, Okay, what kind of tokens am I going to suggest? So more of these tokens will then be rejected. **[00:04:45 · 0]** Unfortunately, there are more and more coding tasks, more and more structured tasks that we see out there. So for those use cases, multi token prediction works exceptionally well. And then you have efficiency on, let's see, knowledge, size, speed. What about complexity? Oh, did I see this one here? **[00:05:08 · 0]** Let me remove that. Sort of it up. And then we have complexity. Because that's also a thing, right? If you have a model that's not open, it really doesn't matter that much how complex it is for the end user. **[00:05:20 · 0]** But when you release a model out into the wild for folks to use, let's make sure it's not too difficult to actually understand, use and process. And so as a consequence of that, there was this model that was released called an encoder free model, and the encoder free model kind of looked at the encoder for processing these multimodal inputs and said, that's a lot of work, what if we remove them? For many of these models in the Gemma four line, you have an audio tokenizer, an audio encoder, and the same for images. And that's great! That works really well. **[00:05:58 · 0]** It's a staple in the field, right, to have these encoders for different multimodal entities, but they still take up a reasonable size. It reduces time to first token, and it's something else to consider when, for example, fine tuning your model. With the Encoder Free model, they try to just remove them entirely and put a lot of the burden of processing these multimodal inputs onto the model rather than these encoders. For audio, it works a little bit like this. You have an input, a sequence of amplitude values, and the the only thing that happens is they cut it up into pieces, make it a sequence of, well, audio tokens, and then project it onto the dimensionality of the model itself. **[00:06:45 · 0]** And that's it. So the only parameters that you have are kind of in the linear projection, but that's about it. And then all of the contextual stuff that was happening in the audio encoder before is now being handled by the LLM itself, so you're kind of shifting all of these parameters from the encoder to the LLM. The same can be done with the image encoder. The one problem though is with images is three d information. **[00:07:14 · 0]** We can't just cut it up into pieces, make a sequence out of it, and just hope the model processes it well because it doesn't, I can promise you that. So what we need for that are positional information. So instead of doing a direct projection from the patches onto the model, we take those patches and provide it with additional positional information. So it knows that, well, let's say patch four, that small image token, has actually some meaning to it. Because you can imagine if you have a very wide image, position four means something very differently than if you have a very tall image. **[00:07:51 · 0]** And so what happens is you just use the positional embeddings, project it, and what you have again is an encoder that's entirely removed. What's so nice about something like this is that it reduces time to first token that very much helps you put all of the capabilities onto the model. And you reduce a big portion of the parameters because especially at smaller sizes, something like 300,000,000 parameters or 500,000,000 parameters. It doesn't sound large when we're talking about hundreds of billions of models of parameters for some of these models, But at 2,000,000,000 sizes or 12,000,000,000, this makes a big, big difference, especially if you can use it for the capabilities of the model. And so we have knowledge, we have size, we have speed, we have complexity, all different ways to focus on efficiency. **[00:08:47 · 0]** There was one last thing that I really wanted to focus on and explain a little bit more about what it means and how it works, and that's architecture. Regular models, regular LLMs are all memory bound and those have advantages and disadvantages. And they're great. They're used for many different purposes and many different fields. What if we make them compute bound instead? **[00:09:13 · 0]** What if we flip the narrative 180 degrees and try to approach it from a very different perspective? And so what you get is not autoregression, it's something entirely different, diffusion. Can we use diffusion for LLMs and still have it being meaningful? Have it used in a way that a lot of people can still use it, but for very different use cases? Now let me go through a little bit about what diffusion means in large language models, especially because we will come to a subject that I think a lot of folks will have seen in the last couple of weeks. **[00:09:52 · 0]** You saw an input, right? We have a query, an input that's being processed, and normally one model will do that and now the same model does it. But what happens in diffusion, we have something called an encoder, and the encoder in diffusion general at least, is still just a decoder with causal attention. It's a fine tuned Gemma four model. No crazy different architecture or what have you, it's one of the models you saw before. **[00:10:20 · 0]** The reason why it's called an encoder though, because it's meant for processing the prompt, and so it generates a KV cache, the context that we need to then build upon and slowly and more surely do inference. So we start with the KV cache. But to do the actual processing to actually generate tokens in the fusion, you don't start from scratch. You don't just know. Here's one token and here's the second and here's the third. **[00:10:47 · 0]** Now you start with a noisy canvas. It's typically sized around two fifty six tokens, and then the job of the denoiser will be to actually process that and iteratively fill it up with tokens that it's very confident about. It is the exact same model though. It uses the exact same, it's tight weights, Jamf4 26 b model. There's one difference though. **[00:11:14 · 0]** If you have a large canvas and you want to update the tokens simultaneously at the beginning and at the end, you kind of need a different type of attention. So when it's in the denoiser mode, it uses bidirectional attention instead, so it can look both directions. What happens after that is it gets the kv cache of the encoder because again it was meant to process the state and get it ready for the denoiser to do its work. And then the denoiser can predict or suggest what tokens should go at which specific places because we have this initial canvas of tokens, and all it needs to do is update them at the appropriate places. So there's this predicted canvas that it then has. **[00:11:58 · 0]** But now it doesn't do stuff perfectly at once, so it needs a couple of steps to do that. There are some tokens it's very confident about, obviously, and some not so much. So what you then get is a so called accepted canvas. Some tokens are accepted and some tokens are not. The tokens that are not accepted, though, those are renoised because the entire purpose of this model is to denoise it, to remove the noise from the original canvas and find the actual tokens that should be there. **[00:12:31 · 0]** And so we end up with a renoised canvas and it can go ahead and do its second step. And now in the second step, it again uses the KV cache and it can use this KV cache to again do a similar processing. But what we found if if you were to do just this, it's not a perfect way to approach this because in the very first step, it had an initial ID of what it wanted to do. Even though some of the tokens weren't accepted, it still had a direction it wanted to go into. And so there's kind of a skip connection there, a self conditioning where you take the logits from the previous one and feed it to the input of the second step. **[00:13:11 · 0]** And then you can do this process over and over again, typically eight steps or something, where it updates this canvas. And then in those eight steps you get two fifty six tokens. And that makes your model go incredibly fast when served for a single user on a larger GPU. Efficiency takes a very different meaning than when you compare compute bound to memory bound, but there are still many use cases for that. One we didn't quite expect would blow up in the last week or so, which is obviously JAF. **[00:13:49 · 0]** Because JAF being a foundational decision model, essentially a foundational classifier that's being used to make decisions, happens to work quite well with diffusion. And it works a little bit like this: Let's say you have a situation: the printer is on fire. Anything that requires you to want to take an action you describe the state, the situation, what have you. You again take that encoder and you process it for the purpose of creating a kv cache. What then happens is instead of having a fully noisy canvas, you kind of pre fill some of these tokens. **[00:14:33 · 0]** And the pre fill are then related to the actions that you want the model to make a suggestion on. Okay, printer is on fire, and your options are to either decide if I should evacuate or call IT. Those tokens don't change. But the others are still noisy. That's the one it needs to fill in. **[00:14:53 · 0]** And what happens is that when you do that, it can make a prediction for only those two tokens and using the things that we saw before make a probability of whether it should do something or shouldn't do something. Just one denoising step. Technically, it can do more, but if you want to make fast quick decisions, this is all it takes to create well, apparently it's called Diffusion Gemma Jeff now. But that works extremely well because nobody fine tuned that model. The only thing that happens is just limited in a way. **[00:15:30 · 0]** But that showcases also this concept of efficiency because it's okay to limit certain things in certain ways because it opens up the way to make sure it can be used for things we didn't imagine before. When you combine all of that together, one thing that makes all of this happen is that it's all about the community, right? Efficiency is not for us, it's for you. It's for the folks that use actually these models. And whether that's on mobile, on a laptop, on bigger GPU, That doesn't matter. **[00:16:05 · 0]** When you release these open models and you have seen a bunch of talks already today talking about that, it needs to be useful for you. That's the main purpose of trying to give back to the community. But in the same way, when you do that, you can see the community giving back to you. Because a decision model based on a model that wasn't necessarily intended for that, well, that's very cool to see. That gives a lot of innovation from a field that, you know, we didn't do. **[00:16:36 · 0]** But by giving people the option and the opportunity to play around with these models, to open them up, to see what's happening inside them, to tweak it, to change it, You get these fun, cool models that who knows where it's going. I've no clue. It's now a big hype, I think for good reason. But the thing is with hypes, we'll see how far it gets. And when you then talk about efficiency, there's all these different types of fields and perspectives that you can focus on. **[00:17:09 · 0]** And I could stand here and talk for hours upon hours on what efficiency means alongside performance. That's not an easy thing to do. Right? There are so many of these incredible open models out there to try to focus on these things. Because it's not just about performance, but also on whether you can actually run the model. **[00:17:31 · 0]** Because it doesn't matter how good a model is, if you can't run it, it's meaningless. And so a lot of attention from different parties, and you see a lot happening in Jammer, is being put onto making them useful on very different devices. And hopefully this helps you get an idea of what it means to focus on these smaller models. It's a very different narrative than going bigger and bigger and bigger. That's also interesting, gives a lot of very cool capabilities. **[00:18:01 · 0]** But these open models, these small models, especially when they're on device on edge, your small devices, they need to be capable and they need to be quick and efficiency from many different perspectives is a very big part of that. Thank you very much. Thank you. **[00:18:30 · 1]** All right. Oops. Okay. Alright. Thank you, guys. **[00:19:02 · 1]** Let's give it up one more time for Martin. Alright. So our next speaker is gonna come in a few minutes. In the meantime, I have so many questions for you guys. So, I just would like to know, what was your favorite part of this conference? **[00:19:26 · 1]** Like, does anybody want to share? Okay. Maybe to open ended questions don't work in a crowd... Big crowd like this. Okay. **[00:19:39 · 1]** So, you guy... I I don't know if you guys went to, to the expo. You checked the swag. You checked, like, but but I, like, I just want to know... For example, for me, what I like the most is, seeing people at the expo discussing, and and and making new friends. **[00:19:55 · 1]** I think it's something that is quite obvious to say. Right? Like, every time we go to a new place, every time we go to an event is to meet people. But I really like that it really becomes like a family. I've been attending AI engineer for quite a while now, around the world. **[00:20:11 · 1]** So in San Francisco, in New York, London, and here. And it's so cool to be able to see, like, sometimes we see the same faces over and over, and then we make new connections. So my my advice, to be honest with you guys, I know we we don't have a lot of time left, but, yeah, just make sure that you connect. Make make sure that, you you collaborate because I think that the... Through this community, like, a lot can happen and you never know. **[00:20:37 · 1]** It doesn't have to happen now. It can happen in the future. Another nice moment that I saw is, so many people lining up for it to talk to, to our speakers. Like, yeah, so many people come to me and they ask me also about, some some of our speakers. Apparently, like, yeah, but p... **[00:20:56 · 1]** Piano AI was quite popular. Arm also is quite popular. So it's good to see that, our audience is, is quite diverse. Nobody wants to share what their favorite moment was? The food. **[00:21:14 · 1]** Alright. Yeah. I I guess so. Yeah. I guess so. **[00:21:17 · 1]** Yeah. Thank you, Sarah, again, for for sponsoring for the food. So, apparently, we nailed that. Okay. Any any talk? **[00:21:23 · 1]** Favorite talk? Gemophora was great. Okay. Wow. Nice. **[00:21:31 · 1]** What what else? Okay. That was Gemma four. That's it. Like, we should have just had Gemma four for the entire conference, and that's it. **[00:21:40 · 1]** The Kitsa, that was hilarious. Man, I love Kitsa. Yeah. Shout out to Kitsa. But, yeah, he's, he's hilarious. **[00:21:47 · 1]** Yeah. I agree. I agree with that one. Alright. Okay. **[00:21:52 · 1]** So enough of me, but this is our last presentation of the day. Okay? So thank you so much for hanging in there. But before we get started, I would like to introduce, Olivier Tebow from Gradium, who is gonna talk about the missing layer of conversational AI. Let's give it up for Olivier. **[00:22:30 · 1]** Okay. Alright. **[00:22:40 · 2]** My book here. Extrabatch Gmail. Oh, that's not what I meant. Nope. Nope. **[00:23:03 · 2]** Nope. Nope. Nope. Nope. It's not what I meant either. **[00:23:07 · 2]** Alright. Should we get started? Let me see if that is working. That is working. Cool. **[00:23:15 · 2]** Alright. Should I get started? Yes? Alright. So let's get started. **[00:23:20 · 2]** Hi, everyone. My name is Olivier Taboul. I'm a CTO and co founder at Gradium. And today, I'm going to talk about the missing layers of conversational AI. It's a mysterious title, so we'll see what that means. **[00:23:35 · 2]** Maybe before we do, let me introduce you, to who we are at Gradium. So, we are a one year old startup, focused on training, voice models. So those acronyms you might have seen if you don't... Did not. STT means speech to text, TTS text to speech. **[00:23:57 · 2]** And S2S, speech to speech. So all these kind of flavors of models, we train and serve at Gradium. Maybe before that, if I go back to to the genesis of Gradium, is QTie. So maybe you've heard of it? I don't know. **[00:24:17 · 2]** Yes? Some of you? Great. So, QTie is a Paris based open science lab that was founded in 2023 with quite some money and a lot of freedom. And out of this freedom, they, you know, made some great scientific breakthrough. **[00:24:34 · 2]** Moshe is probably the most famous and I'll go back to that a bit later in the presentation. And of obviously some others. But the, I would say the success of those voice models were so big that somehow we thought in order to get the, you know, bigger reach of our research idea, we should turn that into products. So the goal of Gradium was really to bridge this gap from research to product. And and here we are, one year later. **[00:25:03 · 2]** So conversational AI. Right? Where where do we get that? Actually, today pretty much everywhere. You've heard about voice agents. **[00:25:14 · 2]** Whether it's in gaming, live streaming, assistance, robotics. Wherever you want to talk to a machine, there is a Voice Agent involved. And I would like to explain you a bit of the history, today. Not only, but a part of it. And if I go back to, well, what I could call provocatively the prehistoric time of voice agents, meaning before elements. **[00:25:42 · 2]** You probably remember that. Right? I think that was the closest, I would say glance at AGI back in the days. Right? It really felt like AGI, right? **[00:25:55 · 2]** That was the iPhone and Siri. And Siri was nothing really AGI, it was a very nice product, engineering product made of different layers, lots of layers. Six of them, sorry. First one is, the speech to text, the speech recognition. After the audio is turned into text, it was turned into, you know, classifiers trying to say what is the intent, what is the object, what is the subject, what is the complement, stuff like that. **[00:26:27 · 2]** A dialogue manager. Some calls to APIs like, you know, check the weather, check the calendar on the phone, stuff like that. Probably the most, you know, I would imagine template and regexes intensive part was to generate some text, for Siri to be spoken out. And then the TTS would speak it out loud. Right? **[00:26:54 · 2]** Then what came after was LLMs. And I say, maybe we don't need all of that, right? This natural language understanding, those templates, all gone. LLM. There's a bit of a lie in here. **[00:27:08 · 2]** Right? There is still a dialogue manager in modern, voice agents. And those LLMs, they become stronger. Right? And they are now able to connect to the external world. **[00:27:21 · 2]** They can interact. They can talk to a database, a web search, MCPs, tool calls, retrieval, rag, and they became voice agents. And that's what you're seeing here is actually how, I would say, 99% of voice agents are working. It's called the cascaded model, ASR or t t... STT, LLM, TTS. **[00:27:47 · 2]** Right? They are not trained jointly. Those are three pieces of AI modules that we plug one into the other. There is a... An extension of this, which is... **[00:27:59 · 2]** Okay. Maybe we don't need three pieces. Maybe we can choose one big model. And that's we call speech to speech. And then there is no turn taking to take into account. **[00:28:10 · 2]** The model itself knows where to speak and when to listen. So this is to give you an overview of what is conversational AI. So I can also give you a glimpse of what we do. And then I'll talk about the engineering aspect and the AI aspect. So what's under the hood? **[00:28:30 · 2]** So what do we do at Gradium? We basically do all of this, right? We do the text to speech, the speech to text, we do the text to speech on device, we do some speech to speech translation, We do some voice design because when you wanna say something out loud, you want to say it in some voice. It can be a French accent voice, for example, or it can be a female voice, high pitch, low pitch, whatever. And coming soon is actually the speech to speech. **[00:29:03 · 2]** So this talk is called, the missing layer of voice AI. So what is actually missing? It feels like it's it's complete. Right? And some people will try to convince you this picture is complete, but it's not. **[00:29:17 · 2]** And those are just some examples. It's not everything. But we're gonna talk more about that. So if you put, you know, voice AI in the wild, well, most likely it's gonna go wild. Right? **[00:29:28 · 2]** Because most of the, the applications do they are more on the phone, one speaker, you know, one computer talking to one another. But put that in the real life, you get, like, many speakers. It confuses the agent. Noises confusing the agent. Long sessions, you leave the... **[00:29:47 · 2]** Lose the context. And also the real global scale. And what I mean real global scale is like not the abilities for anyone to reach your your service, but like the ability as ## Slides ### 00:00:00–00:00:20 # Lookup Table ## Decoder Architecture - Token Embedding Layer - Nx - RMSNorm - Local Attention or Global Attention - RMSNorm - RMSNorm - FFNN - RMSNorm - Per-Layer Embeddings (PLE) - RMSNorm - LM [3D block diagram representing a lookup table with dimensions for Vocab and Layers] [Flowchart diagram illustrating a neural network decoder architecture with various layers and connections] ### 00:00:40–00:01:00 ### Mobile Quant ## Mobile Quants (quantization-aware training) ### 00:01:20–00:01:40 # Mobile Quantization of Gemma 4 (E4B) - Token Embedding Layer - 2 -bit [Diagram showing a large blue rectangular block representing a model, with a "Token Embedding Layer" at the top, and an arrow indicating a 2-bit output from this layer.] ### 00:01:40–00:02:00 ## Mobile Quant ### Mobile Quantization of Gemma 4 (E4B) [Diagram showing a neural network decoder block. It starts with a Token Embedding Layer, leading to a Decoder. Inside the Decoder, there are RMSNorm layers and an attention block labeled "Local or Global Attention". An arrow points from this attention block to two components: "KV Cache" (labeled 8-bit) and "Weights" (labeled 4-bit).] ### 00:02:00–00:02:20 # Mobile Quantization of **Gemma 4** (E4B) - Token Embedding Layer -> 2-bit - Decoder (42x) - RMSNorm - Local or Global Attention - RMSNorm - FFNN - RMSNorm - Per-Layer Embeddings (PLE) -> 2-bit - RMSNorm - LM Head [Diagram showing the architecture of the Gemma 4 model with quantization bit-widths for different layers and components. It highlights 2-bit, 4-bit, and 8-bit quantization for embeddings, weights, and KV cache.] ### 00:03:00–00:03:20 # RMSNorm # LM Head [Diagram showing interconnected blocks labeled "RMSNorm" and "LM Head", with four colored squares between them, likely representing a neural network architecture.] ### 00:03:20–00:03:40 # Target Model - Actions - Embedding Layer - Decoder 1 - ... - Decoder n-1 - Decoder n - RMSNorm - LM Head - speak # Draft Model - Embedding Layer - down - Decoder 1 - Decoder 2 - Decoder 3 - Decoder 4 - RMSNorm - LM Head [Two flowcharts side-by-side comparing a "Target Model" and a "Draft Model". The Target Model shows a sequence from "Actions" through an "Embedding Layer", multiple "Decoders" (1 to n), "RMSNorm", and an "LM Head" to "speak", with "Local" and "Global" inputs connecting to specific decoders. The Draft Model shows a similar structure but with a "down" step after the Embedding Layer and four distinct decoders (1-4), with "Local" inputs to Decoders 1-3 and a "Global" input to Decoder 4.] ### 00:03:40–00:04:00 ## Target Model - Input sequence: Actions, speak, louder, than, pens - Model Architecture: - Embedding Layer - Local Decoder 1 - ... - Local Decoder n-1 - Global Decoder n - RMSNorm - LM Head - Output Evaluation: - louder (correct) - than (correct) - pens (incorrect) [Diagram of a neural network architecture showing an Embedding Layer, a stack of Decoders (some local, one global), RMSNorm, and an LM Head, processing an input sequence. Below the main diagram, three tokens are shown with checkmarks for "louder" and "than", and a cross for "pens".] ### 00:04:00–00:04:20 # Target Model Input sequence: Actions, speak, louder, than, pens Model Architecture: - Embedding Layer - Local Decoder 1 - ... - Local Decoder n-1 - Global Decoder n - RMSNorm - LM Head - Output: words Prediction Example: - 2 tokens accepted, 1 predicted - Tokens: **louder** (accepted), **than** (accepted), words (predicted) [Diagram showing a neural network architecture with an Embedding Layer, multiple Decoders (labeled Local and Global), RMSNorm, and an LM Head, processing input tokens to produce output words.] [Box illustrating a prediction step, showing "2 tokens accepted 1 predicted" with "louder" and "than" marked as accepted, and "words" as predicted.] ### 00:04:20–00:04:40 # Target Model - **Input tokens:** Actions, speak, louder, than, pens ## Model Architecture - Embedding Layer - Local Decoder 1 - Local Decoder n-1 ### 00:04:40–00:05:00 # Target Model Input tokens: Actions, speak, louder, than, pens Output: 2 tokens accepted 1 predicted - louder - than - words [Diagram of a neural network model architecture with an Embedding Layer, multiple Local Decoders, a Global Decoder, RMSNorm, and an LM Head, processing input tokens to output words.] [Box illustrating the prediction process, showing two tokens "louder" and "than" marked as accepted, and "words" as the predicted token.] ### 00:05:00–00:05:20 **Knowledge** • **Size** • **Speed** ### 00:06:00–00:06:20 ### Image Processing Pipeline - Input: Image frames - Repeated 27 times: - RMSNorm - Global Attention - RMSNorm - RMSNorm - FFNN - RMSNorm - Pooling - Linear Projection ### Audio Processing Pipeline - Input: Audio Tokenizer - Repeated 12 times: - FFNN - Local Attention - Convolution - FFNN - RMSNorm - Linear Projection [Two parallel flowcharts illustrating neural network architectures. The left chart shows an image processing pipeline starting with image frames, passing through RMSNorm, Global Attention, FFNN, and Pooling layers. The right chart shows an audio processing pipeline starting with an audio tokenizer, passing through FFNN, Local Attention, Convolution, and RMSNorm layers.] ### 00:06:20–00:06:40 # Model Architectures and Parameters - **~150M Parameters** (E2B and E4B) - **~550M Parameters** (26B and 31B) - **~300M Parameters** [Diagram showing two neural network architectures side-by-side. The left architecture, for image input (depicted by a flamingo image), consists of 27 repeated blocks, each containing RMSNorm, Global Attention, and FFNN layers, followed by Pooling and Linear Projection. The right architecture, for audio input (depicted by a waveform and "Audio Tokenizer"), consists of 12 repeated blocks, each containing FFNN, Local Attention, Convolution, FFNN, and RMSNorm layers, followed by Linear Projection.] ### 00:06:40–00:07:00 # Gemma 4 12B (encoder-free) - amplitude values (640) - 40ms - Audio Tokenizer - FFNN - Local Attention - Convolution - RMSNorm - Linear Projection [Diagram illustrating an audio processing pipeline, showing an audio waveform segmented into 40ms chunks, processed into amplitude values, then through an Audio Tokenizer, a 12-layer block (FFNN, Local Attention, Convolution, FFNN, RMSNorm), and finally a Linear Projection.] ### 00:07:00–00:07:20 ### Gemma 4 12B (encoder-free) [Diagram showing audio amplitude values (640) segmented into 40ms chunks, processed by a Linear Projection into a sequence of embeddings.] [Diagram of a neural network architecture starting with an Audio Tokenizer, feeding into 12 repeated blocks containing FFNN, Local Attention, Convolution, and RMSNorm layers, followed by a final Linear Projection.] ### 00:07:20–00:07:40 + ooling ear Proje [Diagram showing a process flow with interconnected blocks. An arrow points to a circular block containing a plus sign. Rectangular blocks are labeled with partial text " ### 00:07:40–00:08:00 ## 12B - **27x** - RMSNorm - Global Attention - RMSNorm - (Skip connection) - RMSNorm - FFNN - RMSNorm - (Skip connection) - Pooling - Linear Projection ## Gemma 4 12B (encoder-free) - **1x** - LayerNorm - Linear Projection - LayerNorm - (Skip connection with Positional Embedding input) - LayerNorm - Pooling - Linear Projection [Two block diagrams illustrating neural network architectures, both starting with a tiled image of a flamingo. The left diagram shows a 27-layer block structure, while the right diagram shows a single-layer block structure for Gemma 4 12B, including a "Positional Embedding" input.] ### 00:08:00–00:08:20 ### Gemma 4 12B (encoder-free) **Audio Processing Pipeline** - Input: Audio waveform segments (40ms each) - Linear Projection - Output: Sequence of feature blocks **Gemma 4 12B Model Architecture** - Input: Image (1x) - LayerNorm - Linear Projection - LayerNorm - (Addition of Positional Embedding) - LayerNorm - Pooling - Linear Projection - Output: Sequence of feature blocks [Diagram showing an audio waveform being processed by a linear projection layer] [Block diagram illustrating the architecture of the Gemma 4 12B (encoder-free) model, starting with an image input and progressing through various neural network layers] ### 00:08:20–00:08:40 ### ...ear Project [A diagram showing a horizontal timeline or process bar with a segment labeled "40s", two adjacent rectangular segments (one dark green, one light green), and three dots indicating ### 00:08:40–00:09:00 **Knowledge** • **Size** • **Speed** • **Complexity** ### 00:09:20–00:09:40 - Knowledge - Size - Speed - Complexity - Architecture ### 00:10:20–00:10:40 # Input Query - The - LLM - picks - Token Embedding Layer - Encoder (decoder with causal attention) - Gemma 4 (26B A4B) [Block diagram showing an input query processed through a token embedding layer and an encoder (decoder with causal attention) leading to Gemma 4] ### 00:10:40–00:11:00 ### Noisy Canvas - pen - sea - test - hi ### Input Query - The - LLM - picks - Token Embedding Layer - Encoder (decoder with causal attention) [Diagram showing two input streams, "Noisy Canvas" and "Input Query", feeding into a token embedding layer and an encoder with causal attention, represented by stacked bars.] ### 00:11:00–00:11:20 ### Noisy Canvas - pen - sea - test - hi - Token Embedding Layer - Denoise (decoder with bidirectional attention) - LM Head ### Input Query - The - LLM - picks - Token Embedding Layer - Encoder (decoder with causal attention) Gemma 4 (26B A4B) [Diagram showing two processing pipelines: "Noisy Canvas" and "Input Query". The "Noisy Canvas" pipeline consists of input tokens, a Token Embedding Layer, a Denoise block (decoder with bidirectional attention), and an LM Head. The "Input Query" pipeline consists of input tokens, a Token Embedding Layer, and an Encoder block (decoder with causal attention). An arrow labeled "Gemma 4 (26B A4B)" points from the output of the "Noisy Canvas" pipeline to the input of the "Input Query" pipeline.] ### 00:12:00–00:12:20 # Noisy Canvas - Input tokens: pen, sea, test, hi - Token Embedding Layer - Denoise (decoder with bidirectional attention) - LM Head - "predicted" canvas output tokens: all, words, film, once # Input Query - Input tokens: The, LLM, picks - Token Embedding Layer - Encoder (decoder with causal attention) **Shared Component:** - KV-cache (connected from Denoise and Encoder) [Diagram illustrating a neural network architecture with two input branches: "Noisy Canvas" and "Input Query". Both branches start with a Token Embedding Layer. The "Noisy Canvas" branch proceeds through a Denoise block (decoder with bidirectional attention) and an LM Head, outputting predicted tokens. The "Input Query" branch goes through an Encoder block (decoder with causal attention). Both the Denoise and Encoder blocks feed into a shared KV-cache.] ### 00:12:20–00:12:40 # Noisy Canvas - `pen`, `sea`, `test`, `hi` - Token Embedding Layer - Denoise (decoder with bidirectional attention) - LM Head - "predicted" canvas: `all`, `words`, `film`, `once` - Entropy Bound Sampler - "accepted" canvas (accept or reject): `all`, `words`, `test`, `hi` - Entropy Bound Sampler - "re-noised" canvas: `all`, `words`, `dogs`, `Gemma` # Input Query - `The`, `LLM`, `picks` - Token Embedding Layer - Encoder (decoder with causal attention) - KV-cache [Diagram illustrating a neural network architecture with a "Noisy Canvas" processing path and an "Input Query" path, showing token embedding, denoising/encoding, and an entropy bound sampler for canvas prediction and acceptance.] ### 00:12:40–00:13:00 ## LM H... ### 00:13:00–00:13:20 ### Denoising Process Diagram **Step 1** - **Noisy Canvas** (tokens: pen, sea, test, hi) feeds into: - Token Embedding Layer ### 00:13:20–00:13:40 # Noisy Canvas - `pen`, `sea`, `test`, `hi` - Token Embedding Layer - **Step 1** - Denoiser (decoder with bidirectional attention) - LM Head - "re-noised" canvas - `all`, `words`, `dogs`, `Gemma` - Token Embedding Layer - **Step 2** - Denoiser (decoder with bidirectional attention) - LM Head # Input Query - `The`, `LLM`, `picks` - Token Embedding Layer - Encoder (decoder with causal attention) [Diagram showing a two-step denoising process for a "Noisy Canvas" interacting with an "Input Query" encoder via KV-caches and a self-conditioning loop.] ### 00:13:40–00:14:00 # A Two-Step Denoising and Self-Conditioning Process ## Step 1 - **Noisy Canvas** (tokens: `pen`, `sea`, `test`, `hi ### 00:14:00–00:14:20 ### Noisy Canvas - `pen` - `sea` - `test` - `hi` ### Input Query - `The` - `LLM` - `picks` ### Step 1 - **Denoiser** (decoder with bidirectional attention) - Token Embedding Layer - LM Head - **Encoder** (decoder with causal attention) - Token Embedding Layer - KV-cache ### "re-noised" canvas - `all` - `words` - `dogs` - `Gemma` Jev? [Diagram showing a machine learning architecture with a Denoiser and an Encoder, processing input tokens from a "Noisy Canvas" and an "Input Query" to produce "re-noised" output tokens.] ### 00:14:40–00:15:00 # Model Architecture - **Prefill** input tokens: `call`, `IT`, `:`, `hi`, `evacuate`, `:`, `@` - **Input Phrase**: "The printer is on fire." Both paths feed into a **Token Embedding Layer**. The "Prefill" path continues to: - **Denoise** (decoder with bidirectional attention) - **LM Head** The "Input Phrase" path continues to: - **Encoder** (decoder with causal attention) A **KV-cache** connects the Denoise and Encoder components. [Diagram illustrating a language model architecture with two main processing branches: one for prefill tokens and another for an input phrase, showing token embedding, denoising (bidirectional attention), and encoding (causal attention) components, connected by a KV-cache.] ### 00:15:00–00:15:20 # Structured Prediction ## Denoise Model - Input tokens: `call`, `IT`, `:`, `hi`, `evacuate`, `:`, `@` - Token Embedding Layer - Denoise (decoder with bidirectional attention) - LM Head - Output tokens: `IT`, `:`, `evacuate`, `:`, `YES` ## Encoder Model - Input sentence: "The printer is on fire." - Token Embedding Layer - Encoder (decoder with causal attention) - KV-cache (connected to Denoise) [Diagram illustrating two neural network architectures: a Denoise model with bidirectional attention processing input tokens to produce output tokens, and an Encoder model with causal attention processing a sentence, with a KV-cache connecting the Encoder to the Denoise model.] ### 00:15:20–00:15:40 ## DiffusionGemma-Jev "The printer is on fire." P(no) 70% P(yes) 30% [Diagram illustrating a model architecture with a Denoise block (decoder with bidirectional attention) processing an input sequence ("call IT : hi evacuate : @") and outputting a modified sequence ("call IT : NO evacuate : YES") along with probabilities (P(no) 70%, P(yes) 30%). An Encoder block (decoder with causal attention) processes the input "The printer is on fire." and feeds into the Denoise block via a KV-cache.] ### 00:15:40–00:16:00 IT NO [Flowchart diagram with down arrows connecting labeled boxes. A grey rectangular box is labeled "IT", and a green rounded rectangular box is labeled "NO".] ### 00:16:00–00:16:20 # All About The Community ### 00:17:20–00:17:40 - **Knowledge** - **Size** - **Speed** - **Complexity** - **Architecture** ### 00:18:00–00:18:20 - Knowledge - Size - Speed - Complexity - Architecture ### 00:18:40–00:19:00 - AI Engineer Paris - Backblaze - CLOUDFLARE - Cockroach Labs - CodeRabbit - cominty - Daytona - descope - FACTORY - FIMO - langfuse by ClickHouse - Latitude - MAKORA - Mistral - Modal - neo4j - Notion - NVIDIA - Qdrant - SENTRY - SIERRA - Sonar - stripe - ZenML labs [Grid of company and event logos] ### 00:19:00–00:19:20 - NVIDIA - AI Engineer Paris - Cockroach Labs - Mistral - FACTORY - AI Engineer Paris - SENTRY - Sonar - CLOUDFLARE - Daytona - AI Engineer Paris - langfuse by ClickHouse - Mistral - Modal - AI Engineer Paris - neo4j - Mistral - Qdrant - SENTRY - AI Engineer Paris - Sonar - Cockroach Labs - NVIDIA - AI Engineer Paris - AI Engineer Paris - FACTORY - CodeRabbit - Mistral - AI Engineer Paris - cominty - descope - Mistral - Latitude - AI Engineer Paris - Mistral - NVIDIA - FIMO - AI Engineer Paris - MAKORA - stripe - ZenML labs - Backblaze - AI Engineer Paris - Notion - SIERRA - Cockroach Labs - AI Engineer Paris - FACTORY - SENTRY - Sonar - Mistral - AI Engineer Paris - NVIDIA - CLOUDFLARE - Daytona - AI Engineer Paris - AI Engineer Paris - Mistral - langfuse - Sonar - AI Engineer Paris - Modal - neo4j - Qdrant - CodeRabbit - AI Engineer Paris - NVIDIA - cominty - descope - AI Engineer Paris - Latitude - MAKORA [Grid of company logos and names] ### 00:19:20–00:19:40 - AI Engineer PARIS - Notion - SIERRA - Mistral - AI Engineer PARIS - NVIDIA - langfuse by ClickHouse - Sonar - AI Engineer ### 00:19:40–00:20:00 NVIDIA, Artingeering, Cockroach Labs, Mistral, FACTORY, SENTRY, neo4j, CLOUDFLARE, Daytona, langfuse, Modal, Odr ### 00:20:00–00:20:20 ### Partners & Sponsors - NVIDIA - AI Engineer Paris - Cockroach Labs - Mistral - FACTORY - SENTRY - Sonar - CLOUDFLARE ### 00:20:20–00:20:40 - AI Engineer PARIS - Notion - STE - Mistral - AI Engineer PARIS - NVIDIA - langfu by Cli - sonar - AI Engineer PARIS - NVIDIA - cominty - desc(*) [Grid of company logos and text, some indicating "AI Engineer PARIS" roles] ### 00:20:40–00:21:00 - Backblaze - AI Engineer PARIS - Notion - Sonar - Mistral - NVIDIA - cominty [Logos of various companies and organizations] ### 00:21:00–00:21:20 - AI Engineer PARIS - Notion - Mistral - AI Engineer PARIS - NVIDIA - langfuse by ClickHouse - sonar - AI Engineer PARIS - NVIDIA - minty - desc [Grid of company logos and names] ### 00:21:20–00:21:40 - AI Engineer PARIS - Notion - Mistral - AI Engineer PARIS - NVIDIA - langfus by Click - Sonar - AI Engineer PARIS - NVIDIA - mint ### 00:21:40–00:22:00 - AI Engineer PARIS - Notion - Mistral - AI Engineer PARIS - Sonar - lang - NVIDIA - cominty [Grid of company logos and names] ### 00:22:00–00:22:20 - AI Engineer PARIS - Mistral - Sonar - NVIDIA - Cominty - Notion - Backblaze - LangChain [Logos of various ### 00:22:20–00:22:40 # AI Engineer PARIS Organized by Mistral STATION F | September 23-24 # Olivier Teboul CTO [Mistral logo] [Gr logo] ### 00:22:40–00:23:00 - NVIDIA - AI Engineer Paris - Cockroach Labs - Mistral - FACTORY - AI Engineer Paris - SENTRY - Sonar - CLOUDFLARE ### 00:23:00–00:23:20 # Join us Try us for free gradium.ai github.com/gradium ### Share 'AI Engineers (Gradium)' - People with access: Olivier Teboul (you ### 00:23:20–00:23:40 # The Missing Layers of Conversational AI Olivier Teboul CTO & Cofounder Gradium [Abstract illustration of intertwined, dotted purple and green lines] ### 00:23:40–00:24:00 # Who is Gradium? Our mission is to **unlock the unrealized potential of voice AI**: fluid, natural and reliable voice as the new interface for AI. We train voice models: STT, TTS, S2S. [Photo of a group of people in an industrial-looking room] [Abstract graphic of scattered colored dots] ### 00:24:00–00:24:20 ### Who is Gradium? Our mission is to **unlock the unrealized potential of voice AI** fluid, natural and reliable voice as the new interface for AI. We train voice models: STT, TTS, S2S. [Photo of a group of people standing in an industrial-style room] [Abstract visualization of many small, scattered colored dots] ### 00:24:20–00:24:40 # From Research to Production ## Kyutai: Breakthrough Research Paris-based open science AI lab. Founded 2023 with €300M. Team from Meta FAIR, Google DeepMind, and Inria. - **Moshi** Speech-native dialogue system. - **Hibiki-Zero** Real-time STS translation. - **Pocket-TTS** CPU model for TTS ## Gradium: Scaling Impact Built to bring research to production. $70M raised in 2025. [Logo for moshi.chat featuring a green concentric circle design and a vertical bar graph-like element] ### 00:24:40–00:25:00 .chat /kyu ### 00:25:00–00:25:20 # From Research to Production ## Kyutai: Breakthrough Research Paris-based open science AI lab. Founded 2023 with €300M. Team from Meta FAIR, Google DeepMind, and Inria. - **Moshi** Speech-native dialogue system. - **Hibiki-Zero** Real-time STS translation. - **Pocket-TTS** CPU model for TTS ## Gradium: Scaling Impact Built to bring research to production. $70M raised in 2025. moshi.chat by /kyutai [Logo with concentric green circles and a series of vertical bars, some filled green, some red, some empty] ### 00:26:00–00:26:20 # Anatomy of SIRI 1. **Audio Input** - User speaks (push to talk) 2. **Speech-to-Text (ASR)** - Converts speech to text 3. **NLU (Intent + Entities)** - Classify intent and extract slots - `Intent: SetTimer` - `Duration: 10 minutes` 4. **Dialogue Manager (State Tracker)** - Tracks conversation state and decides next action 5. **Skills / Services (APIs)** - Call the appropriate service or API - Alarm / Timer - Weather - Calendar - Messaging - ...More Services 6. **Response Generation** - Generate response using templates and fill slots 7. **Text-to-Speech (TTS)** - Convert text response to spoken audio 8. **Audio Output** - Siri speaks the response [Flow diagram illustrating the 8 steps of Siri's anatomy, from audio input to audio output, with an icon for each step.] [Examples of intent and duration slots for NLU.] [List of example services/APIs for Siri skills.] ### 00:26:20–00:26:40 # Anatomy of SIRI 1. **Audio Input** User speaks (push to talk) 2. **Speech-to-Text (ASR)** Converts speech to text 3. **NLU (Intent + Entities)** Classify intent and extract slots - Intent: SetTimer - Duration: 10 minutes 4. **Dialogue Manager (State Tracker)** Tracks conversation state and decides next action 5. **Skills / Services (APIs)** Call the appropriate service or API - Alarm / Timer - Weather - Calendar - Messaging - ... More Services 6. **Response Generation** Generate response using templates and fill slots 7. **Text-to-Speech (TTS)** Convert text response to spoken audio 8. **Audio Output** Siri speaks the response [Flow diagram showing 8 steps in the Siri process, each with an icon: microphone, sound waves, magnifying glass, menu icon, grid of squares, quotation marks, speaker, and sound waves.] ### 00:26:40–00:27:00 # Anatomy of SIRI 1. **Audio Input** * User speaks (push to talk) 2. **Speech-to-Text (ASR)** * Converts speech to text 3. **NLU (Intent + Entities)** * Classify intent and extract slots 4. **Dialogue Manager (State Tracker)** * Tracks conversation state and decides next action 5. **Skills / Services (APIs)** * Call the appropriate service or API * Alarm / Timer * Weather * Calendar * Messaging * ...More Services 6. **Response Generation** * Generate response using templates and fill slots 7. **Text-to-Speech (TTS)** * Convert text response to spoken audio 8. **Audio Output** * Siri speaks the response [Diagram illustrating the 8-step process of Siri, with each step represented by an icon and connected by arrows. Under step 5, a list of example services is shown. A small UI element for setting a timer is partially visible near step 3.] ### 00:27:00–00:27:20 C... NLU [Diagram showing a box labeled "NLU" and a partial label "C..." above it, connected by green lines.] ### 00:27:20–00:27:40 # Voice agents ## 1. Audio Input User speaks (streaming audio) ## 2. Speech-to-Text (ASR) Streaming transcription ## 3. LLM Agent Reason -> act -> observe, looping over tools before it replies - Agent loop - Reasoning / Planning - Tool Calls (APIs) - Memory / Context - Retrieval (RAG) ## 4. Text-to-Speech (TTS) Streaming spoken audio ## 5. Audio Output Agent speaks the reply [Flow diagram illustrating the five steps of a voice agent: Audio Input, Speech-to-Text (ASR), LLM Agent, Text-to-Speech (TTS), and Audio Output.] [Icon for Audio Input: a microphone.] [Icon for Speech-to-Text (ASR): a sound wave.] [Icon for LLM Agent: a robot head with a waveform inside.] [Icon for Text-to-Speech (TTS): a speaker volume icon.] [Icon for Audio Output: vertical sound bars.] [Diagram within LLM Agent showing "Agent loop" connected to four components: Reasoning / Planning, Tool Calls (APIs), Memory / Context, and Retrieval (RAG).] ### 00:27:40–00:28:00 # Agent Loop - Reasoning / Planning - Tool Calls (APIs) - Memory / Context - Retrieval (RAG) [Icon representing a network or hierarchy next to "Reasoning / Planning"] [Icon representing a wrench and screwdriver next to "Tool Calls (APIs)"] [Icon representing a stack of discs (database) next to "Memory / Context"] [Icon representing a magnifying glass next to "Retrieval (RAG)"] ### 00:28:00–00:28:20 # Speech-to-speech LLM ## 1. Audio Input User speaks (streaming audio) ## 2. Speech-to-Speech LLM Listens and generates audio in one model — full-duplex, listening and speaking at the same time **Full-duplex - no turn-taking** Replaces the entire cascade: - ASR - **LLM** - TTS ## 3. Audio Output Agent speaks (streaming audio) [Diagram illustrating a three-step process: Audio Input, Speech-to-Speech LLM, and Audio Output, with the LLM step replacing a cascade of ASR, LLM, and TTS.] ### 00:28:20–00:28:40 # What does Gradium do ? [Abstract ring-like shape composed of many small green and blue dots] ### 00:28:40–00:29:00 # Our Models ### Text-To-Speech API Streaming, multilingual, and high-fidelity voice cloning and voice design for robust AI agents ### TTS On Device: Gradium Phonon On-device. Optimized to run on CPU with your chosen voices. Total privacy with zero latency. ### Speech-To-Text API Streaming and multilingual ### 00:29:00–00:29:20 ### Speech-To-Text API Streaming and multilingual transcription powered by semantic VAD for natural turn-taking. ### Speech-To-Speech API Real-time translation generation and ### 00:29:20–00:29:40 # Voice AI **in the wild** - Noise - Long sessions - Many speakers - Global Scale [Illustration of a humanoid robot in a futuristic, industrial setting] ### 00:29:40–00:30:00 # Voice AI in the wild - Noise - Long sessions - Many speakers - Global Scale [Image of a humanoid robot in an industrial environment]