# From research to production: How we rebuilt a streaming model for Voice AI — Hervé Bredin — session 2026-09-24T10:00:00.000Z → 2026-09-24T10:30:00.000Z _95 transcript lines · 55 slides_ ## Transcript So you can see that In 2026, we have this crazy increase of free trial views. Because, like, if you are having success, like, you're actually also an attractive target on the Internet. What's good news, though, is that it's a solvable problem. Like, just as you can see an exponential growth of the attacks, you can also see a very strong and steep decline of the attacks if you have the right solution. Just in four months, we blocked, like, 3,000,000, attacks like this for free trial. So that's definitely part of top of mind for us. And then, like, of course, there's another vector. You you might be familiar with pay as you go pricing. So pay as you go abuse goes with that. So what happens is that the consumer would basically consume, let's say, a thousand or thousands of dollars of tokens. Month. And then at the end, you try to get the money, and the card is just invalid. So it's kind of like, you know, Michelin star, like we say in French, but four four tokens. Right? So by the time you know the compute is gone and you can't charge anything to that user. So pay as you go is an amazing tool for pricing, but you have to think about how to manage fraud in that case. So the companies handling that well are monitoring usage Signals in real time and make sure that they see all the different signals, and and avoid that from happening. Okay. So what do you do about this? Like, three things. Again, please monitor this at every stage of the life cycle from account creation to post usage. So defend all of this for your customers. Then I would recommend to close the gap between usage and billing. Like, you don't want to wait a month to bill You want to make sure that you close that gap as much as possible. And then, like, please build fraud into your launch plan. Like, it's not necessarily the best or more most exciting part about building, but that's actually incredibly important. Like, at every step of your product, you need to think about, okay, if I am a bad actor now and that I want to take advantage, what does the attack looks like? And so in my view, if you haven't thought about this or you don't Have the answer yet. Like, you're not ready to ship it. So the playbook is being rewritten. We all know that. But, like, how you built, how you go global, how you price, how you adapt your go to market motions, and how you manage fraud. That's what's changed, not just the tools, but actually the whole starting line of building a company. And you you know that you can build in hours. You can be present in 42 countries on Day one, as we can see. You can price for value. So that's actually, in our view, you know, the funniest and maybe craziest time of, like, building, for for building a company in the world. So we we hope that these insights give you some some idea of, like, how to build this. And, yeah, like, our job at Stripe is to be there with you to help you scale. So, I'm really excited to see what you build. And just please Know that we are there if you need to scale as well, and drop me a message if you want to chat more. Thank you. Alright. Thank you, Ariel. Let's give it up for Ariel once again, please. Alright. Okay. I think what, the the fact that, all the information, the piece of information that, stayed with me through this presentation is that you need to design for humans, but you also need to design your For agents because they might, you know, be purchasing your products as well, which is I think it's awesome. I asked earlier a question. I said, how many people are builders here? And many of you raised their hands. Right? Like, yeah. Most of you are builders. Right? Like, you're building products, and you're, maybe chatting with agents. My personal experience is that I kinda stopped, typing to my agent. So I only chat mostly with voice, and I find it, like, a a more natural way. To interact with agent. I read faster than I than I hear, but I also I I talk a lot faster than I than I than I than I type. I don't know if anybody shares that experience here. Who talks to their agent? Yeah? Okay. So it's actually a good number. Well, our next speaker is here to talk to us actually about voice AI and real time. So without further ado, please join me in welcoming to the stage chief scientist and cofounder of Pianote AI, Herve Sit down. Okay. Let me try this thing. Yeah. It's working. Hi, everyone. Thank you. For the introduction. It's the first time I've been, ever announced on a stage by an MC. I feel like a bit like, Celineo in Paris, tonight, or, today at least. Don't worry. I won't sing anything. But I will, talk to you about voice AI and, in particular, about speaker diarization. So, a quick introduction about myself. So I'm the cofounder and chief science officer at Pianote AI. I've been an Research most of my career up until two years ago when we, created this company. And, so, yeah, I've been mostly focusing on speaker diarization for the last fifteen years. And, it looks like people think that, I know a few things about speaker diarization, and I hope today that, I'll tell you what speaker diarization here is and, that you'll learn a bit about how that works and, yeah, that at least you you take something out of this talk. So what is speaker diarization? So speaker diarization, basically, is the task of taking a recording of a conversation between multiple speakers, and outputting some metadata about it. What you can do, starting from a conversation, is basically do voice activity detection first, where you would, detect, the parts where, someone is speaking and when nobody is speaking. That's voice activity detection. The second step that you can do Top of that is, splitting those, very long, speech tone regions into, single speaker regions. That's this task is called segmentation. So it includes both speaker change detection as well as overlapping speech detection. So in this example here, in in the middle, you see that, they are, it looks like there might be, at least two speakers because, there's someone interrupting, another one or starting speaking after The end of the, another speech turn. And a bit later in the conversation, there is a very small speech turn that overlap with the other one. This is these are usually what we call back channels, like, mhmm, okay, yes, that you say when you are nodding at someone, talking to you. And those events are very important to, for voice AI agents to, basically understand really the core of the conversation. The words are usually not enough, knowing that when someone is interrupting someone else or nodding at what they are saying, it gives A lot more information about the conversation. And the next step, really, what we call speaker diarization, is then once the conversation is segmented into speech turn, is to actually assign each speech turn to the right speaker. And this is done without knowing in advance the number of speakers, so that makes the task more difficult than usual, supervised machine learning task. And we don't know the identity as well of the speaker. We have to, basically generalize to any speaker, even those that Are not part of any training sets. So that's what makes, speaker diarization difficult. And so there are many tasks, where speaker diarization can be useful, and lots of those tasks are can fall actually under the knowing who said what is just as important as what was said. So, for instance, for video dubbing application, when you want to start from a video in French, for instance, and you want, basically to put it On the Internet and reach a larger audience, an international audience, then you can do use video dubbing, tools to, basically translate from French to other languages and making sure that the, you know, the conversation remains natural. So you have to put the right, speaker clone, voice to on the right speaker. And for that, you need to know exactly who speaks when in the original audio. Other tasks, obvious tasks are meeting note takers, especially in hybrid meetings where Some people are joining from the same meeting room. When there are multiple streams, speaker diarization not necessarily, is useful. But when people are in the same room, that's where it's it's really useful. And this allows to to make some nice, summaries of the conversation of who's supposed to do what after the meeting, for instance. And other applications, like podcast intelligence, where you want to track one particular speaker in multiple podcasts or track the host of a podcast across episodes. And So we have this, open source toolkit called, PianoTodio, that, allows you, actually, to get from a raw audio recording to, an actual speaker diarization. So it's become very popular over the years. I have some vanity metrics here. Thanks to our friends at Hugging Face, it's also very visible there. It has, on average, 10,000,000 downloads every month. And basically, in a few lines of code, you can get from the audio to The actual speaker diarization. So here in this example, we start, for the first block of the code, as basically downloading, the Community one model, the open source Community one model from Hugging Face. You get this, object called Community one here in the Python code. And then you simply apply, this, pipeline on a recording of a conversation, and you get the prediction that you can then iterate over to know exactly start time, end time, and the actual speaker time. And this really, on in Three lines of code. Yeah. So this, that I described was batch in the sense that, the whole processing, happened after the conversation took place. So why, go real time? So there are many, reasons why we we would want to go real time. For voice agent, it's not that obvious because usually there's, one person talking on an or to an agent, and, you you don't really care about, speaker authorization because there's only One speaker. But very soon, and it's already there, actually, the agent will listen to to the conversation while we're having a meeting, for instance, than to a use case would be to flag some speakers interrupting another speaker too often or to flag a speaker talking too much. And, doing that in real time is necessary. And, the next step is also that we soon are going to have robots in our house and to basically answer to a child or To the adult in the in the household, you would, basically need to know who's who. This is really where, real time voice diarization speaker diarization is useful. So now that we know that it's actually useful, how, did we at Peanut AI, manage to go real time? I need first to take a step back to explain how the batch speaker diarization that I just described earlier in three lines of code called Community one works. So, I'm going to spend some time on this slide a bit more. So, basically, given a, let's say, a one hour conversation, a recording of a one hour conversation, how it works, how Community One works today is we start by splitting the conversation into pieces. Let's say twenty seconds or thirty second pieces. So each green rectangle here in this slide, actually, represents thirty seconds of speech, let's say. You see that they are overlapping a bit. So We we move that in a in a sliding, manner, in a sliding window manner. And then we apply the first, neural network that basically performs speaker diarization on each chunk separately. So why do we do that? Why do we process the conversation pieces by pieces rather than trying to do that globally? It's because working on chunks allows to have a limited number of speakers as output, so the machine learning task is actually Easier and safer to to solve like that. So basically, the model is trained to take thirty seconds of audio as input and returns, basically, a matrix of, let's say, the time and the number of speakers. And it will output zero and once whether each speaker is active over time. And the problem with this kind of first approach is that at that point, the same speaker on the first chunk might have a different label on the same for the in Different chunk. So we need to rely on a second step, where each speaker in each chunk here represented by symbols, the triangles, the circles, and the stars, are projected into a high dimensional space using speaker embeddings. I'm pretty sure you're all familiar with what embeddings are, but basically, those are high dimensional representation of a thing. Here, the thing is a speech turn. So each speech turn is projecting into this space. Here, it's Represented as three d, but in practice, it's oops. In practice, it's coming back. Is it? It is. I should probably stay here. And the speaker modeling model, it's a neural network as well. It's trained to have two speech terms of the same speaker close to each other, and two speech turns from two different speakers far away from each other. And this way, we can actually find a cluster. In this space. And this is how, basically, we managed to reconcile all the, the chunks, finally and be able to basically stitch, all those small speech done together and and get the final results. So that's the starting point of, of speaker diarization in a batch manner. So the question is, how did we get from, this, batch, diarization to an actual, real time implementation? So the first thing that we, worked on is basically the clustering part that I just mentioned. And instead of having it global, having it incremental. So, this, animation will will say it also. So basically, starting with a conversation between two speakers here, in batch, diarization, we start by extracting speaker embeddings from all those speech terms. So you get them. All those points are actually speaker embeddings. And you simply do clustering. Into that. But if you turn to streaming, you don't have access to the future because the conversation did not happen yet. So the idea is that, the clustering algorithm needs to work incrementally. And that makes its job way harder because you only have a limited view of the conversation. So, you see here, this is a fake example, but it takes some time for the clustering algorithm to realize that there is actually two speakers. And that is why it makes, you know, switching from batch to Real time quite difficult if we if we were to stay with the, you know, chunk wise approach that I just mentioned. It's still managing the end, to find the right clustering, but you saw that during, the actual conversation, it made some mistakes. And so with this kind of approach, this is actually implemented in the Die Hard open source toolkit, which is a creation of our CTO, Juan Manuel Correa. And we managed to Basically, get slightly higher diarization error rate, at around five second latency, because the chunks are around five seconds, long, or at least the steps between the chunks are five seconds. So we said, okay, that's great. But the next step would be to manage to lower this diarization error rate at least to be as good as a batch speaker diarization. And also, this five second latency might be a bit too much. So, what we worked on next is They're basically, turning this chunk wise segmentation into something about more streaming that we call causal segmentation, where instead of waiting for the whole chunk to be available to process it, basically processing it in streaming. So that that brings, a lot more, of other difficulties. So, let's focus, for instance, on speaker chain detection. In batch mode, what we usually have access to the whole conversation. So speaker chain detection can be Done like that. You slide a window, around the course of the conversation. You compare what happens to the left of the the segment and to the right, and you compute the difference. And you get some nice peak where there's a speaker change. But in streaming, the problem is that you don't have access to the future, so it takes some time for the model to actually reach, you know, confidence enough, to, enough confidence to basically take a decision. And this small amount of time, that it needs in the future to Decide whether there was a speaker change or not. We call that the look ahead. And that's basically a latency that we can't get rid of because if we at one particular point of time in the conversation, you need to decide whether there is a speaker change, unless you have a very strong modeling of the prosody or something like that. You really have to wait whether someone else is starting speaking to actually take the decision. So that brings this, incompressible, latency. That is added to the to the thing. And, so we we went ahead and and implemented that. And we, with, basically, lots of research on the on the research team on this particular segmentation streaming model. We managed to actually, using a much larger model than the chunk wise one, and also some, some implementation of loss function. I won't go into the detail. I am happy to discuss that. So that we managed to basically reach the same level of, accuracy as the batch speaker diarization with a very low latency of four hundred milliseconds. And this four hundred milliseconds actually, involves many things. It it involves also a round trip from, the client to the servers and back. So I'd like to spend now some time on the actual forty hundred... Four hundred milliseconds latency. So what's in those four hundred milliseconds latency? So We start by a client, having a conversation locally. The the the first latency that you get is basically that the conversation is split into hundred millisecond blocks. So you already have to wait one hundred millisecond, let's say at time t. You have to wait one hundred milliseconds to build this block. So already at time t, we already have one hundred millisecond latency. This latency that I put here, is part of the four hundred millisecond latency, and it's not something that is often Counted in the numbers that you can see here and there. But it's actually, also something, that is, important to take into account. Then, the next step, basically, is we are relying on a Cloudflare durable object technology to basically send this block size to the closest Cloudflare server. So this, on average, takes twenty milliseconds, but it really depends on where the client is and how good Cloudflare is, basically. And on this server, what we do is we receive many streams, and we batch them. So we batch lots of one hundred millisecond blocks so that, we can process multiple streams in parallel. So this is very fast. And then once we've built those batches, we send them to a model where the GPU horsework lives. So basically, for segmenting those one hundred millisecond latency, a batch size, let's say, of 32 Two streams. Let's say that's just an example here, but it takes around, fifty milliseconds just for the inference. And, also, you have to take into account this two hundred milliseconds look ahead that I mentioned earlier. So, basically, when you reach time t, what the model the neural network does, it predicts what happened two hundred milliseconds in the past. So this is, this adds up. And then in parallel, we find this trick of actually running the embedding extraction and the clustering in a totally async. Manner. So this does not delay the prediction whatsoever, so that's why I I put it, zero milliseconds here. And then we do the the the round trip back. So five more milliseconds here. Then, we basically, using the state that is stored in the CloudFlare, we basically stitch the pieces together and serialize it to send it to the client. So in total, you get those four hundred milliseconds. And if we go into more details, so basically, they are just not just one latency. Basically, there are many latencies here. There is the network latency that, basically, really depends on the client, how close, the client is to the networks, how fast, their connection is. So that accounts for, like, fifty milliseconds on average, but this can be moving parts. Then there's the, inference latency that I mentioned. The 50, milliseconds. So same. This inference latency, you can tweak it a bit, obviously. If you make a higher batch, a larger batch, it will take more time. But then, your cost will be lower because you can process, more streams, with the same, with the same GPU. But if you want to be faster, then you have to lower the batch size, and your cost will go up. So really, this inference latency, this is where you Control a cost, latency trade off. And then there is this algorithmic latency that takes basically most of the, the latency here. And same. It controls a new trade off. This is a trade off between accuracy and latency. The more you can, wait in this look ahead, the better the prediction you can make. So this is really, like, as I say, a a trade off between the two. An easy way for to to to reduce the latency would be to reduce the blocks. Nice, obviously. Here it's one hundred milliseconds. But that has repercussion on, basically, the inference latency because you get way more blocks that you need to process. And so, yeah, that makes the whole trade off a bit more difficult to achieve. So, yeah, no, I've now reached the point where I think it would be good to make a demo. So, I'm going to ask my co founder Vincent to join stage. Just so that we have a chat quickly. And you can take a mic here, and, I'll switch, to a demo so that you you get really what four hundred milliseconds really means in practice. So just about the setup. Even though we have two microphones here, it's really my laptop that captures, that captures the, the conversation. So, So you you're revealing the tricks. You say everything. Yeah. Hi, everyone. Nice to meet you. So how did my talk go? Quite good. I hope Well, you share the and people have been learning things about what we do. I'm always amazed how difficult it is. It sounds easy when you use it. But Yeah. But we can even, do overlapping speech detection. That's what we usually do. Yeah. 20% of the speech usually is overlapping, which is kind of crazy to detect. Yeah. He keeps interrupting me in in every meeting we have. Oh, definitely. It's a nice effect. For some reason, Raouf says something in the in the backside because we found speaker speaker zero two here. Please don't backchannel, please, Raouf. Okay. I'll leave you there. Thanks, So so you you get the idea of what, basically four hundred milliseconds feels like. And, yeah, so I'm I I I've reached, almost the end of my talk. I would like I hope you you you learned a few things in this, in this talk. And, at least, the the three things that are, I think interesting you'll tell me, about this talk is that, I I tried to convince you that, batch and real time diarization are actually two different machine learning problems. They have very different constraints, even though the input and the output are the same. Solving it needs different approaches. I wanted to highlight as well that latency is just not one number. It really depends on many things. I mean, you all knew about that. But in particular, that for speaker diarization, you see that the main latency is coming from the algorithmic part because of the way this speaker diarization task is set up. And, yeah, very proud of the work that the research and tech team did to handle this. And, yeah, there are many ways of trading between latency, accuracy, and cost. And, yeah. So thank you very much. That's the end of my talk. And, yeah, happy to have a chat after if if you want. Thank you. Thank you so much, Harvey. Let's give it up one more time for Harvey, please. Nice. Alright. Okay. So this is a good moment because this is this was the last talk of this Morning. Okay? So please, you can go to the expo. We have a launch that is being sponsored by Sierra. So shout out for to Sierra, please. Yeah. Thank you for offering us some fuel, you know, to, to to get going with this, with with this event and to come back stronger in the afternoon because we still have so many things to talk about. We're gonna be back here at 2PM. We're going to have a fireside chat between Mistral and NVIDIA. You should not Is this at all. So I'll see you here at 2PM. Alright? Enjoy your lunch. ## Slides ### 00:00:21 # Multi-account abuse **1 in 6** sign-ups at AI companies involve multi-account abuse ### 00:00:50 # Free trial abuse ## Impact of Radar abuse prevention [Line graph showing a metric over time, with a sharp increase peaking around October 2025 labeled "Adoption", followed by a significant decrease and stabilization at a lower level, illustrating the impact of Radar abuse prevention.] ### 00:01:22 # Pay-as-you-go abuse [A display showing a dollar sign and the amount 0,000.00] ### 00:01:49 ### Pay-as-you-go abuse [Illustration of a stack of cards with a prominent button on top displaying a red cross icon and the text "No payment"] ### 00:02:23 ## Checklist #5: Monitor fraud risk - Defend every stage of the lifecycle. - Close the gap between usage and billing. - Build fraud into your launch plan, not your post-mortem. ### 00:02:49 ### Checklist #5: Monitor fraud risk - Defend every stage of the lifecycle. - Close the gap between usage and billing. - Build fraud into your launch plan, not your post-mortem. ### 00:03:21 # Adapt go... # M...or fra... ### 00:03:50 - Adapt go- - Monitor fr ### 00:04:21 - NVIDIA - AI Engineer Paris - Cockroach Labs - Mistral - FACTORY - SENTRY - Sonar - neo4j - CLOUDFLARE ### 00:04:52 - Mistral - langfuse by ClickHouse - NVIDIA - PARIS - AI - des [Logos of Mistral, langfuse, and NVIDIA] ### 00:05:19 # Partners / Sponsors - AI Engineer Paris - Notion - Mistral - langfuse - Sonar - NVIDIA - Cominty ### 00:05:47 # AI Engineer PARIS Organized by Mistral ## STATION F September 23-24 # Hervé Bredin Chief Science Officer, Co-Founder pyannoteAI [Logo for AI Engineer Paris event] [Logo for Mistral] [Logo for pyannoteAI] ### 00:06:47 # From batch to realtime ## speaker diarization in the era of voice AI Hervé Bredin pyannoteAI > David Cameron > @dave_cameron > `@hbredin` makes `pyannote` and I'm fairly certain it's the gold standard in speaker identification > Atila > @atiorh > This is why we partnered > `@hbredint` [Illustration of three head silhouettes (two grey, one green) above a green waveform, representing speaker diarization] [Illustration of a goat] [Trophy emoji next to a social media post] [Hand holding an object emoji next to a social media post] ### 00:07:22 # What is speaker diarization? Recording of a conversation No prior knowledge on - the number of speakers - their identity [Waveform representing a sound recording] [Three timeline diagrams illustrating voice activity detection, segmentation (including speaker change & cross-talk), and speaker diarization over time] ### 00:07:50 # What is speaker diarization? ### Recording of a **conversation** ### No prior knowledge on - the **number** of speakers - their **identity** ### voice activity detection ### segmentation (including speaker change & cross-talk) ### speaker diarization [Green waveform representing a sound recording] [Three timeline diagrams illustrating the process of speaker diarization: the first shows detected voice activity, the second shows segmentation including speaker changes, and the third shows identified speakers (green and black blocks) over time] ### 00:08:19 # What is speaker diarization? Recording of a conversation No prior knowledge on - **the number of speakers** - **their identity** [Audio waveform representing a conversation] [Three timeline diagrams illustrating the process: voice activity detection, segmentation (including speaker change & cross-talk), and speaker diarization, showing different speakers identified by alternating colors over time] ### 00:08:49 # What is speaker diarization? Recording of a conversation No prior knowledge on - the number of speakers - their identity voice activity detection segmentation (including speaker change & cross-talk) speaker diarization [Green waveform representing a sound recording] [Three timeline diagrams illustrating the process of voice activity detection, segmentation, and speaker diarization, showing blocks of detected speech and identified speakers over time.] ### 00:09:18 # Knowing who said what is just as important as what was said ## Video translation - speaker-consistent voice cloning ## Meeting note taker - multi- vs. single-channel ## Podcast intelligence - track hosts & guests across podcasts [Diagram showing a video reel with a French flag translating into video reels with Spanish, Italian, and American flags] [Diagram showing a video conference screen connected to a person's silhouette, which then connects to a checklist document] [Diagram showing three podcasts, each with multiple speaker silhouettes, illustrating how a specific speaker (highlighted in green) can be tracked across different podcasts] ### 00:09:53 # Knowing **who said what** is just as important as **what was said** ## Video translation speaker-consistent voice cloning ## Meeting note taker multi- vs. single-channel ### 00:10:25 ## Product #1 ## Product #2 [Diagram showing a grid of human silhouette icons, arranged in columns labeled "Product #1" and "Product #2". Some figures are dark grey ### 00:10:55 ```python uv add pyannote-audio # download community-1 from Huggingface from pyannote.audio import Pipeline community1 = Pipeline.from_pretrained( 'pyannote/speaker-diarization-community-1' ) # apply community-1 on conversation prediction = community1('conversation.wav') # iterate over speaker-labeled speech turns for (onset, offset), speaker in prediction.speaker_diarization: # {speaker} speaks between {onset} and {offset} ... ``` - 10K+ stars - 10M+ monthly downloads - 300K+ users [Box with GitHub logo and emoji next to usage statistics] ### 00:11:21 # pyannote.audio: Speaker Diarization Example ### Installation `uv add pyannote-audio` ### Usage Example ```python # download community-1 from Hug ### 00:11:44 # Why go **real-time**? ### 00:12:22 # Why go **real-time**? from voice agent ... to conversational assistant pyannoteAI [Diagram showing the evolution from a single person interacting with a smart speaker (voice agent) to two people interacting with a smart speaker in a more complex, multi-turn conversation (conversational assistant)] ### 00:12:49 # S... # ...ro ### 00:13:23 # Starting point: **community-1** batch pipeline - few seconds sliding window - local "end-to-end" speaker diarization (fixed maximum number of speakers) - stitching with frame-wise majority vote - agglomerative hierarchical clustering (adaptive number of speakers) - speaker embedding [Diagram illustrating a speaker diarization pipeline, showing audio segments processed through local diarization, then stitched and clustered based on speaker embeddings to identify distinct speakers (speaker #1, speaker #2, speaker #3).] ### 00:13:50 # Starting point: **community-1** batch pipeline [Diagram illustrating a speaker diarization pipeline, showing a few-seconds sliding window, local "end-to-end" speaker diarization, speaker embedding, agglomerative hierarchical clustering, and stitching with frame-wise majority vote.] ### 00:14:18 ## Starting point: **community-1** batch pipeline - few seconds sliding window - local "end-to-end" speaker diarization (fixed maximum of speakers) - speaker #1 - speaker #2 - speaker #3 - stitching with frame-wise majority vote - agglomerative hierarchical clustering (adaptive number of speakers) - speaker embedding [Diagram illustrating a speaker diarization batch pipeline, showing a sliding window processing audio segments, local diarization, speaker embedding, stitching with majority vote, and agglomerative hierarchical clustering to identify speakers.] ### 00:14:51 # Starting point: **community-1 batch pipeline** [Diagram illustrating a speaker diarization pipeline. It shows a "few seconds sliding window" processing audio segments, followed by "local 'end-to-end' speaker diarization" to identify speakers within fixed maximum numbers. These diarized segments are used for "speaker embedding". The process also involves "stitching with frame-wise majority vote" and "agglomerative hierarchical clustering (adaptive number of speakers)" to group speaker embeddings into distinct speaker clusters (Speaker #1, Speaker #2, Speaker #3).] ### 00:15:53 # Starting point: community-1 batch pipeline [Diagram illustrating a speaker diarization pipeline. It shows audio segments processed by a "few seconds sliding window" and then by "local 'end-to-end' speaker diarization". The output feeds into "speaker embedding" generation, followed by "agglomerative hierarchical clustering" to identify speakers. The final step is "stitching with frame-wise majority vote" to produce a timeline of speaker segments (speaker #1, speaker #2, speaker #3).] ### 00:16:21 - Speaker 1 - Speaker 2 **BATCH** - cluster all at once **STREAMING** - cluster incrementally [Audio waveform visualization] [Diagram showing two distinct clusters of blue and green dots within a box labeled "BATCH"] [Empty box labeled "STREAMING"] ### 00:16:50 - Speaker 1 - Speaker 2 ### BATCH · cluster all at once ### STREAMING · cluster incrementally CURRENT ESTIMATE **1 speaker** [Audio waveform showing segments attributed to Speaker 1 (blue) and Speaker 2 (green), with a highlighted segment] [Diagram showing two distinct clusters of dots, one blue (Speaker 1) and one green (Speaker 2)] [Diagram showing a small cluster of two blue dots with a question mark above it] ### 00:17:53 # How to go **real-time**? - **From batch community-1** - chunk-wise segmentation & clustering - **Diarization error rate ( ### 00:18:18 # How to go real-time? ## from batch community-1 - chunk-wise segmentation & clustering - Diarization error rate (DER) ~ 20% - Batch processing ## ... to fake streaming diart - chunk-wise segmentation & **incremental** clustering - Diarization error rate (DER) ~ 25% - 5s latency - [juanmc2005/diart](https://github.com/juanmc2005/diart) ## ... to real streaming live-1 - causal segmentation & **incremental** clustering ### 00:18:48 ## STREAMING - Speaker A - Speaker B CURRENT SPEAKER **Speaker A** [Diagram illustrating an audio waveform with 'before' and 'after' sections, and a 'difference' label] [Indicator box showing 'CURRENT SPEAKER: Speaker A'] ### 00:19:17 # STREAMING - Speaker A - Speaker B CURRENT SPEAKER **Speaker B** [Diagram showing an audio waveform with a segment highlighted from time `t` to `t + Δ`, labeled "before" and "after". Below it, a curve illustrates "difference" and "lookahead Δ" representing "algorithmic latency".] ### 00:19:49 ## How to go real-time? - from batch **community-1** - chunk-wise segmentation & clustering - Diarization error rate (DER) ~ 20% - Batch processing - ... to fake streaming **diart** - chunk-wise segmentation & **incremental** clustering - Diarization error rate (DER) ~ 25% - 5s latency - ... to real streaming **live-1** - causal segmentation & **incremental** clustering - Diarization error rate (DER) ~ 19% - < 400ms latency ### 00:20:24 # How to go **real-time**? ## from batch **community-1** - chunk-wise segmentation & clustering - Diarization error rate (DER) ~ 20% - *Batch processing* ## ... to fake streaming **diart** - chunk-wise segmentation & **incremental** clustering - Diarization error rate (DER) ~ 25% - 5s latency ## ... to real streaming **live-1** - **causal** segmentation & **incremental** clustering - Diarization error rate (DER) ~ 19% - < 400ms latency - (major breakthrough by research team) ### 00:20:50 # What's in **400ms**? [Diagram showing a client box with an audio waveform and two person icons, labeled "100ms block size", and a separate green block below also labeled "100ms block size"] ### 00:21:17 # What's in 400ms? [Diagram showing a client sending a 100ms block size of data over a ~20ms network to an orchestrator/CPU] ### 00:21:52 # Orchestrator ### 00:22:20 # What's in 400ms? - **client** - **100ms** block size - ~**20ms** network - **orchestrator | CPU** - batching & state management - <**1ms** - <**5ms** network - **worker | GPU** - segmentation - **50ms** inference - **200ms** lookahead --- **Timeline breakdown:** - **100ms** block size - **20ms** (network) - **50ms** inference - **200ms** lookahead [Flow diagram illustrating a process with three main stages: client, orchestrator, and worker, showing time durations for each stage and network transfers.] [Icon of three people and an audio waveform in the client block.] [Cloud icon in the orchestrator block.] [Green geometric logo in the worker block.] ### 00:22:47 # What's in 400ms? [Flow diagram illustrating a processing pipeline with time breakdowns] - **client** (100ms block size) - `~20ms network` to - **orchestrator | CPU** - batching & state management (`<1ms`) - `<5ms network` to - **worker | GPU** - segmentation: **50ms** inference, **200ms** lookahead - embedding + clustering: **0ms** async [Horizontal bar chart showing a timeline breakdown of 400ms] - **100ms** block size - **20ms** network - **50ms** inference - **200ms** lookahead ### 00:23:18 # What's in **400ms**? - network latency: 50ms - inference latency: 50ms - algorithmic latency: 300ms [Segmented bar chart showing a breakdown of time components: 100ms block size, 20ms, 50ms inference, 200ms lookahead, and 20ms] ### 00:23:46 # What's in **400ms**? - **network latency**: 50ms - depends on the client - **inference latency**: 50ms - controls cost / latency tradeoff - **algorithmic latency**: 300ms [Diagram showing a breakdown of 400ms into network, inference, and algorithmic latencies with their durations and dependencies] [Horizontal bar chart showing a timeline breakdown: 100ms block size, 20ms, 50ms inference, 200ms lookahead, 20ms] ### 00:24:22 [Partial view of a bar chart or timeline with numbers '20' and '5'] ### 00:24:50 [Horizontal bars, one grey with "20", one blue with "5", and one green] ### 00:25:24 ### Diarize live audio in real time [Screenshot of a web application interface for live audio diarization, showing a cURL code example] ### 00:25:52 ## Precision-3 is live! - 10.4% lower diarization error, same pricing. ### Precision-3 [Icon resembling a gear or atom] ### 00:26:25 # What's in 400ms? - **network latency**: 50ms - depends on the client - **inference latency**: 50ms - controls **cost / latency** tradeoff - **algorithmic latency**: 300ms - controls **accuracy / latency** tradeoff [Flow diagram breaking down 400ms into network, inference, and algorithmic latencies with their durations and associated factors] [Partially visible timeline-like graphic at the bottom showing ### 00:26:50 # Take-home message - Batch & realtime diarization are two different problems. - Latency is not just one number (network + inference + algorithm). - Research and tech teams worked hand-in-hand to find the optimal trade-off between latency and accuracy. ### 00:27:19 - ...premium - ...on - ...ealtime dia... - ...or rate - ...arization - ...19% ### 00:27:50 # Sponsors / Partners - AI Engineer PARIS - Notion - Mistral - langfuse by ClickHouse - Sonar - NVIDIA - cominty - des... [Grid of company logos and names] ### 00:28:17 # AI Engineer ## PARIS ### 00:28:53 # AI Engineer Paris ## Organized by - Mistral ## Platinum - NVIDIA ## Gold - Cockroach Labs - FACTORY - SENTRY - Sonar ## Silver - CLOUDFLARE - Daytona - langfuse by ClickHouse - Modal - neo4j - Qdrant ## Bronze - CodeRabbit - cominty - descope - Latitude - MAKORA - FIMO - stripe - ZenMLabs ## Supporting partners - Backblaze - Notion - SIERRA