# Generating Identity at Scale, ElevenLabs' AI-Driven Orb — Dorian Lods — session 2026-09-24T08:00:00.000Z → 2026-09-24T08:30:00.000Z _74 transcript lines · 57 slides_ ## Transcript If you wanna know more about what we do at Eleven Labs or if you have any questions later, please feel free to note my socials. I'm here today to talk to you about, how in a creative per perspective, things have changed a little bit in the past few years, and how how we deal with all these new tools, set of tools, or AI tools enabled, and so on. So, just let's take, let's let me just take a take a step back and tell you how I used to make the thing and know how I make the thing that makes the thing. Is a bit of a change. So I came from an industry where I used to make immersive creative experience websites. They are mainly based in WebGL or immersive installation or whatnot. They were all designed around a feeling, refined over thousands of hours. Each one had a specific craft and purpose. During that journey, I interacted a lot with how you can streamline stuff with a system, or you Can produce more or just expand your capabilities, with new tools, for example, Comfy UI, which is an amazing tool, that helped creative new immersive, like, new deeper AI systems, but still made for a single outcome, and you will see that it's important for later. And all of this background led me to a specific project I work on, which is this guy. It's OpenAI voice, the face of OpenAI when there he is the voice model. Few notes, but it's it was the first iconic orb that I had to do. Everything was handcrafted, had to fit on brand, and was the goal was to to find a way to visually represent the voice that you talk to, while representing the brand. So pretty much the same stack than before. It was hand tuned, deterministic, everything you see on screen. It's made with a shader. So math actually makes the visuals, all the inputs, all the colors, all the way the thing behaves were carefully tuned, but it was just made for a single purpose. And I spent a few weeks designing one sphere, then I had to design for 40,000,000 voices, which is a big change of scale if you ask me. So I joined ElevenLabs, for the one who don't know us. ElevenLabs is a European first based company. We are leading the way on we have our own frontier in the research team dedicated to any voice surface, any voice interaction, music, any sounds, to be honest. But we do much more than that. We have our own creative platform where you can dub, make new clone your voice, prompt another voice, and so on. So many of our or many many creators. Already used the platform. We have the the web one. We have the app. We also have, 11 agents now. So we handle our customer support call. We do our own orchestration. Many many many things. Here are some key numbers, because I run off numbers, right? We support over 90 languages, we already have millions of business, developers, creators on the platform. We crossed 600 millions of By end of June. Current valuation is at 11,000,000,000 as of February, and now we actually expanded the team to over 700 people across 50 countries. We have more than 10,000,000 agents on the platform. It's a lot of numbers, I know. But some here are some fun ones. So every day, we generate more than thirty four years of audio with our speech to text model, so the scale factor is definitely here. And one We we made 25,000,000 tracks with our music, our music models, and different music touch points. And one that I really really, that I really love, I think it's my favorite number here, is that we so far helped to restore more than 11,000 voices for people who have lost them due to illness or injury. We are super committed towards that goal. Our impact team is doing a stellar job to bring voice capabilities back to the people who actually need it, for free. But more specifically, I joined Eleven Labs Design, so the design team is pretty much all of us here. We try to push the frontier of both the brand, the product, the way you interact with AI, with all our stuff that we have on the platform, and how we can streamline that. I joined with her my first challenge actually, was to explore how voices, which is our main interaction, could have a visual identity and how that Identity could work across the product. So how you bring more soul, visual souls, to our voices and how those souls should reflect the brand. Here's what we've been cooking in the past few months. I might have spoiled a few things with that video. So, the pitch was simple, but yet not so much. We believe that every voice is every face. And then we had 40,000,000 voices on the platform. So, users, they can prompt voice, they can clone their voice. You have we serve many type of different voices, and so they all, they all represent now more than 40,000,000 voices. And how do you design something that is iconic enough, that still translates the voice characteristics, but keeps it in one visual system that each voice needs to bring the Of ElevenLabs, and you still need to fill like ElevenLabs. So it's all about filling and developing it at scale. First thing first, you needed a reframe. So it could not just be me making the outputs anymore, it was me making the system that makes an output, very close to what we've been hearing in the past, yesterday, basically, and today this morning, with our, code factories and so on. But here, more on the design side. The core concept of the arm is actually a stack of two layers. You have the image layer that carries the brand characteristic, the voice that is actually linked to the voice, And then you have the real time layer on the top which brings that to life, add the interactivity, add the living piece to the one in the back. So a bit of a change compared to the first one I worked on at OpenAI. And that touch point then is diffused everywhere across. The board. Let's work it first on the image layer. It's as it core, it's linked to the voice. So one voice equals one face. If you make your new voice, you make a new a new orb. So we need to we went down to the source, and we we analyzed the voice output for air like the audio output for AV voice, we extracted key metrics, key numbers, that that Fall under different category of pitch, energy, expressiveness, and that that drives the rest of the pipeline. Well, at this core, it's very much linked to the voice. In the first place, we wanted to use embeddings directly, but it can became a bit complicated later. To work to design the process and to really expand or to explore how much range do we need to work with, we extracted a set of 2,000 This is the most popular we have on the platform, and we realized, for example, that they are not distributed evenly. You have much more deep voices, mainly for narration purposes, so we really had to focus in those kind of categories, whereas with the other ones might fall under, like, niche, where it's super high, super energetic, like, maybe cartoonish style of voices. But, that really helped us a lot during the process to to explore the thing at scale. First thing first, we wanted to The voice is to feel like the brand. So we needed to teach a model to speak 11 laps. Each voice is unique. The goal is to create a generated new voice per... For a new visual, new... Or per voice. So we needed to find a way so a model can actually learn our design language and how that translate. So off the shelf model, they don't really know your brand. So we built a synthetic dataset to train specific Around the top, and actually there's a pretty fun thing that we explored, is that we steer the inference with the captions. So, on top of the action words, we hand selected multiple types of colours and feelings that we wanted to attribute to different kinds of voices, let's say high pitch, mid pitch, medium pitch, and then we put that in the caption. So during the training, the model could actually Here a bit and, like, develop a bit more, the variety. And all those keywords were actually were put back into inventory. So at inference level, when you rebuild the prompt, the model can choose exactly the same keywords except keywords that has been in the training data. That was really funny. We explore different type of model, one from our friends. So it's a good segue, actually. So we we try to Yeah. We explored different kind of open wide model. The goal was to to find a good sweet spot between, between, like, inferencing cost and development at large scales while it's still maintaining the quality. Post inference, after you generate the base image, the idea is that basically you blur it and you crop it inside the base image, so each variation, each Orb has a really really good variation into it. That's what we call post inference. And that prepares for the rest of the shader. So let's now focus on the real time layer of those orbs. During the design phase, we we actually used code as a core design medium. So it lives as a shader, it takes the image in the back and has multiple layers on top of it. Everything was made through an interface where all the design team was able to tweak. Like, we see how we can feel things. Some parameters are tweaked with the sound itself, with the audio feedback, and so on. WebGL Power is the interactive experience on the platform, on the on the marketing website, but, but we converted this to OpenJAR support for server side for server side rendering for many previews or other touchpoints that cannot actually run the shader themselves. The shader itself is composed of basic Six six stacks together, you have a real time fleet simulation that has, the ripple effect that's linked to the audio. It's really bringing the organicness into it. The goal I might have skipped that. But the goal was to anchor the orbs into nature. Nature is the main characteristics of the orbs. We wanted to feel them them to feel organic. Then you have multiple phases of noise on the top to bring it to life, FBM, and so on. And we have a a third There's sound waves that is actually a link with the one we had before, and the the the brain drain that we put on many of our touchpoints. So all of that led to the shader playground, like I said before, it was the place where most of the crash was happening early on. But behind that, we actually developed a reviewing tool to debug everything at scale. So one voice equals one output, so you also mean that You cannot choose the output, and every output had to be right. So we quickly realized that in order to be able to maintain quality across the board, we needed to reduce it at scale. And that was really a key factor because you might end up with your old debug ones that are super nice, and then as soon as you run it at scale, it actually falls down. So building internal tools to review things at scale really, really make the make a change. That's the less fun part. So I had an we had an amazing, an amazing whole process, an amazing workflows, and then I run through, finance. And they told me that, okay, Lauren, it's super nice, but 40 millions times 10¢, you know, it's way too much. So you need to find a way to reduce both latency and the cost. So we went back to the or to the chart that I shared before where we have all the distribution. Voices in the platform, and we actually, took this as the core source, so instead of generating a new image every time, we generated a gigantic atlas at once, and then from this atlas we actually cropped a bit deeper in each of the images, and so that helps us to not generate every time we we make a new voices, but still maintaining a uniqueness. So every voice is unique, because we rotate it, we crop it differently. We change it with the U and so on, and they are still really much locked into the audio characteristics. That helped to save more than $3,600,000, so pretty good. I ran it 10 to 15 times faster, and have less than three seconds per voice to generate the visual. So it's happening on the back, as soon as you make a new voice and then it runs through all the audio process, it spin ups it spin ups the task on the infra, and And generate the one, so less than three seconds actually make it not visible for the user. And so here's a recap of the pipeline for the voices. You have the voice analysis in the first place that maps key metrics to visual mapping, like a medium pitch, energy level, texture, and so on. Then we have the entire generative pipeline I just showed you that prepared the assets, prepared the shader base, the text. Sure. The preview, the thumbnail, animated m p four, all of this, diffuse the CDN, and so on. And at the end, you have 40,000,000 plus outputs, that serve pretty much everywhere as you can as you're gonna see in a few. Quality development was really, really important. The concept the core design concept was was cool, but we wanted it to feel alive everywhere to have to to to follow all the pipelines of the orbs that we have. The platform. Here you see on the left, you have the the voice cloning, the voice cloning interface, so the orb change depending on your voice. We wanted to maintain quality across iOS, Android. This is the one you see in the middle. Every shader had to look the same. Super important for the brand. And we needed to review it at scale. So on the right side, you have, like, the admin interface where you can You every voice test that we have. All of that started to grow. We made a dedicated service. So we have a dedicated cloud service for making our art visuals now. It's called Picasso because it's a painter, and and I'm French, so it's a shared API across all our touch points. We do we push those orbs everywhere, so in the core app, the marketing web Website on iOS and Android app, Eleven Music, which are which is our music brand branding and expanded into the brand core itself. So we made brand tools, motion designs, everything now used the Orbs, the carrier brand, the carrier voice, and the carrier face. It was backward compatible because we had already previous types of Orbs in the platform, so we were able to run a backfill of 40,000,000 voices. That backfill that backfill ran for a few Two days. Really stressful moment, but it worked at the end. And now they are everywhere. We cannot hold them anymore. They they are part of our, homepage. The home page, the first thing you see on the home page is the orbs. They are part of the app itself, So you see for example here you have deeper voices, it has a tendency to be a bit darker, if they have more energy they will have like blue hints and so on. So I really Invite you to play with the platform and to try to filter voices, to to see how each characteristic of voice translates into visuals. More renders. And they actually came out of the screen, which is really really cool and something I am super proud of. We use them as a core brand element towards everything we do. So, on the left here, it's an out of home campaign we made in The goal, in France, where you could have spot herbs based on the on the on the branding of some of our clients. Here is Alain, for example. Or on the right side, you have London, where we took over one of the main squares of London with our train line, color DNA and so on, so once again another client of ours. And then we the Orb as you said doesn't carry identity but it's a shape shifter, so we can also use it for Expressing, so illustrating the the needs of our clients towards the thing we do. And here, another example, in the Paris metro. It was really fun to go to work and then see that on the metro. So, here is what we have learned during this little process. Using machine learning as a core creative medium changes how you design. It changes the way you think. You don't think about only the outputs, but you need to think about how you Create that. Machine learning, AI tools are basically superpowers, but they need to be used carefully, if if every design thing that you do. Don't overlook it. Now the cost of making the thing has become so much lower that we have tendency to ship fast and fast and fast. And we believe as a design team that the the edge is like, the quality edge is gonna make you actually, want more of brand awareness, brand Quality and so on. So, you need to own the pipeline. You need to own all the outputs like if they were your own. The crash shifted into the data and rules. This is how you're trying to maintain all the AI stuff that we have and to be sure that they follow a branding rule, a filling rule, and so on. Every stage had to preserve the brand vision. Like I said, branding is getting more and more key, and the partner What a single output hit means that we need to debug at scale. If you're making a if you're making a pipeline a process that is meant to to be deployed 40,000,000 times, you need, for example, to to to debug it at scale. So don't you don't hesitate to run multiple run of of generation at a smaller scale to be sure that every you can own every single output. So I still care about every output, like I said. They are all my little They are all I'm all proud of them. There is not anyone that I don't like. But instead of designing them, I designed the condition that produces it. And that's really, really important, and it's something that we've seen on the code side, and we also see it now on the design side. That's pretty much it for me. Thank you for your attention, and if you have any questions, please don't hesitate to reach out, check out our website, our apps, or everything. And, thank you very much. Alright. Thank you so much, Dorian. Let's give it up one more time for Dorian, please. Thank you. Alright. Okay. So, show of hands, who here has, used OpenWAIT's models before? Okay. Cool. So if you did, you probably have used the tools from our next speaker's company. So up next is someone from Hugging Face. He's going to dive deep into how a transformers model loads in V l l m. So, yeah. Hold on to your seats, guys. That's going to it's going to go deep. So please join me in welcoming to machine learning engineer at Hugging Face, Harry Miller. Alright. Okay. So hello, everyone. I'm Harry, and as you just heard, I'm a machine learning engineer at Hugging Face, and I'm also a maintainer of VLLM. And in today's talk, I'm gonna talk to you about how a Hugging Face transformers model can be loaded directly into VLLM and why because of what we do to it, it will run just as fast as if the model had been dedicated had a dedicated implementation in VLL. Itself. But first, a little bit of context in case you haven't heard of VLLM before. VLLM is an open source LLM inference engine. It came from the UC Berkeley Skylab, and is now a PyTorch Foundation project. And the key things that it brings is page attention, which allows you to kv cache efficiently, continuous batching, which allows you to process a request as soon as it arrives rather than waiting for the next Batch to begin. Fast kernels scale out through many different axes of parallelism, portability by running on any hardware, in any environment and then support broad support for hundreds of models, both through dedicated implementations and the transformers modeling back end, which you'll hear more about imminently. And And so then the transformers modeling back end, just again more context, is what lets you Take this transformers modeling code from Hugging Face transformers and run it directly in VLLM. So the model definition, the config, and the weights all come from transformers, and then the runtime and all the performance critical modules come from VLM. So you kind of get a best of both. And so why won't you end up using the transformers modeling backend? Well, there are three main reasons. One of them might be that you're already using the transformers ecosystem, and so you don't To reimplement your model again for each inference engine that you might want to support. So through back ends like this one for VLLM and then also SG Lang, you can take this code that you wrote once for your transformers training loop and run it straight in VLLM or SG line. You might be using it for RL, in which case you would be using traditionally, potentially transformers for your training. And then for rollouts, you would use VLLM, both with Print modeling implementations because previously, that was what you had to do. But now with this, you can use the same code for training and inference during your RL loops, meaning that there's no potential drift in the two d implementations and the training should go a little better. And then finally, it might just be the only way that your model is supported in VLLM. So today, there are 14 architectures which only which are officially support Did via the transformers modeling back end, and that list is only growing as time goes on. So now we start getting into the into the weeds and talking about how we get into the transformers modeling back end and what it does. So obviously, first, we start with starting up a VLLM engine, and here we have a whole load of command line arguments. One of them is to force the transformers modeling back end because for this, QEN3VL model that we're looking at There is a dedicated VLLM implementation, so we have to tell VLLM that transformers is what we want. And then we've used a a not recommended but possible, parallelism scheme where we're using as many as we can across eight GPUs. We've got tensor, pipeline, data, and expert parallel all at once. And then finally, we've enabled torch compile on the multimodal encoder, which not all, the MLM models will do by default, but because of the generality of The transformers modeling back end, if the transformers code, has a compilable multimodal encoder, you can just enable this flag. And if your model works, it should also work via VLLM. So we first cross the boundary into transformers from VLLM by looking into the config where we're looking at what is this model, what shape is it, and then any other features like what quantization scheme is it using. And we then once we've chosen the transformers modeling backend, had a look inside the config, we need to choose which transformers modeling backend class to load our transformers model into. And on the right, you can see the selection of classes available. And the way that we select them is by looking into the config and establishing which characteristics the model has, which will determine the correct class to use. So first, they We'll start with transformers, as you can see. And then we detect this one is multimodal because it has both a text and a vision config, in that, config JSON you saw on the previous slide. It's MOE because it has more than zero experts, and then for cause l l m is the default, if your model is a causal model or if you don't specify that you want to do a pooling task with your ## Slides ### 00:00:26 # Training voice # Understanding ### 00:00:46 I used to make the thing. Now I make the thing that makes the thing. ### 00:01:19 ## Made by hand ### Immersive WebGL experiences Designed to evoke a feeling, refined over thousands of hours. Each one crafted for a specific purpose. [Screenshot of a WebGL experience showing planets and spheres in a cosmic setting] [Screenshot of a WebGL experience showing stylized birds flying through a streaked landscape] [Screenshot of a WebGL experience showing a glowing, textured sphere with the text "I'm generating your unique pioneering aura."] [Screenshot of a WebGL experience showing a light background with a pattern of small dots] ### 00:01:51 ## A system for ## A single outcome Even a bespoke visual starts with a system. I built the tools, tuned the parameters, and refined the result by hand. Every decision served one specific creative direction. [Screenshot of a video editing or 3D animation software interface with a graph and a character on a green screen] [Image of a stylized shoe in a colorful, abstract environment] [Image of a car part surrounded by pink and yellow flowers] ### 00:02:22 GPT Juniper Open and upbeat [Screenshot of a mobile phone interface displaying a circular blue cloud image] [Abstract image with blue and white cloud-like patterns] [Design panel with ### 00:02:53 # AI Engineer ## PARIS Organized by Mistral ### Behind the visual - **Hand-tuned.** - **Deterministic.** ```glsl // compute circle shape vec2 ### 00:03:22 # Voices App Interface - **Add new voice**: Create or clone voice in seconds - **My voices** - **Explore** ## Default voices - Lily: Classy, Social Media ### 00:03:50 ### Voices app ### ElevenLabs: Artificial Intelligence Research [Screenshot of a mobile app showing a list of AI voices, including Lily, Brian, Charlie, Burt Reynolds, and Deepak Chopra] ### 00:04:20 - Support over 90 languages - $600M ARR by end of June 2026 - 10 million agents on our platform - We've helped to restore the voices of 11,000 people who have lost them due to illness or injury - Millions of businesses, developers and creators on the Platform - $11B at Series D in February 2026 - Every day, 34 years of audio is generated with our STT models - 700 people across 50+ countries - 25M tracks created with our music models ### 00:04:49 - Support over 90 languages - $600M ARR by end of June 2026 - 10 million agents on our platform - We've helped to restore the voices of 11,000 people who have lost them due to illness or injury - Millions of businesses, developers and creators on the Platform - $11B at Series D in February 2026 - Every day, 34 years of audio is generated with our STT models - 25M tracks created with our music models ### 00:05:18 # ElevenLabs ## Design At ElevenLabs, the challenge changed. I joined the design team first to explore how voices could have a visual identity, and how that identity could work across the product and the brand. [Photograph of a group of people, likely a team, standing outdoors with mountains in the background] ### 00:06:18 - Karin Slaughter: The Secret We Hid - Homer: The Odyssey - The 10-Hour Work Month by Laura Anderson - Cyrus [A carousel of book covers] ### 00:06:54 ### Screenshot of a Music Discovery Application Interface [Screenshot of a music discovery application interface displaying sections for "Live Sessions," "The Eleven Album, Vol. I," and "Trending Now," featuring album art ### 00:07:19 # Speech Engine - Your agent - Or type a message... [Diagram showing a "Speech Engine" component connected by a line to a "Your agent" component, which has a message input field.] ### 00:07:54 ### ElevenLabs Careers [Screenshot of the ElevenLabs careers page, showing a list of departments with job counts on the left, and a list of open positions with filters and locations on the right.] ### 00:08:22 # IIElevenLabs ### 00:08:53 # 40 million voices ## One visual system Each voice needed a visual identity of its own. Every identity still had to feel like ElevenLabs. [A dark, reflective sphere] ### 00:09:22 M [Partial view of a light green rounded rectangle with the letter 'M'] ### 00:09:47 ## The core ### Concept - **Image Layer** - Voice characteristics - to Base Image - **Real Time layer** - WebGL shader - Thumbnails and Preview [Two images side-by-side: a blurry landscape with red rock formations and blue sky, and a spherical object with swirling red, orange, and blue colors on a black background.] ### 00:10:21 # Reading a voice We analyse voice samples to describe characteristics such as **pitch, energy, and expressiveness**. These descriptors guide the visual identity, translating qualities of the voice into inputs for image generation. ```json { "voice_id": "4za2k0XGgUd57HRSQ1fn", "http_status": 200, "response": { "id": "4za2k0XGgUd57HRSQ1fn", "voice_name": "Lendario - Social Media", "status": "completed" }, "config": { "erb_pitch": 3.76, "pitch_range": 9.54, "pitch_sigma": 3.62, "pitch_velocity": 28.63, "shimmer": 11.1194, "jitter": 2.7905, "hnr": 11.84, "pace": 0.0, "color_scheme": "warm", "pitch_category_level": 2, "pitch_category_name": "Deep", "expressiveness_category_level": 4, "expressiveness_category_name": "Engaging / Vibrant", "energy_category_level": 3, "energy_category_name": "Engaged / Standard", "texture_category_level": 4, "texture_category_name": "Raspy / Gravelly", "pace_category_level": 0, "pace_category_name": "Unknown" } } ``` [Audio waveform and spectrogram showing pitch and energy over time] [Grid of generic audio file icons with alphanumeric filenames] [JSON output of voice analysis results] ### 00:10:50 # Reading a voice We analyse voice samples to describe characteristics such as **pitch**, **energy**, and **expressiveness**. These descriptors guide the visual identity, translating qualities of the voice into inputs for image generation. ```json { "voice_id": "4zAZX0K0Ggds7HR501fn", "http_status": 200, "response": { "id": "4zAZX0K0Ggds7HR501fn", "voice_name": "Lendario - Social Media", "status": "completed", "config": { "pitch": 3.76, "pitch_range": 9.54, "pitch_sigma": 3.62, "pitch_velocity": 28.63, "shimmer": 11.104, "jitter": 2.7995, "hnr": 11.04, "pace": 0.0, "color_scheme": "Warm", "energy_category_level": 2, "pitch_category_name": "Deep", "pitch_category_level": 4, "expressiveness_category_name": "Engaging / Vibrant", "energy_category_level": 3, "energy_category_name": "Engaged / Standard", "texture_category_level": 4, "texture_category_name": "Raspy / Gravelly", "pace ### 00:11:22 # 2,000 voices. A spectrum of expression. ## The shape of our voice collection ### Category Distribution **Pitch** - 1. Very Deep: 4 ### 00:11:47 ## Brand Design R&D Process ### Teaching the model to speak ElevenLabs Off-the-shelf models don't know your brand. We built a custom synthetic image dataset grounded in ElevenLabs' visual language, then trained LoRAs on that imagery to bias generation toward the right aesthetic territory. [Grid of abstract generative art images featuring organic shapes and flowing colors in purple, yellow, green, and orange tones] ### 00:12:23 # Steering inference ## Captioning as a design tool Rather than retraining for each use case, we used AI labeling and custom captioning during LoRA training to steer the model toward specific inference ### 00:12:50 ### Steering inference ## Captioning as a design tool Rather than retraining for each use case, we used AI labeling and custom captioning during LoRA training to steer the model toward specific ### 00:13:20 # Model Exploration We explored different model and LoRA combinations to understand how each interpreted the same visual direction. The challenge was balancing variation with a recognisable ElevenLabs aesthetic. [A grid of images comparing two model combinations: "Claude + Z-Image Model + LoRA" and "Claude + Flux Dev + LoRA". Each combination displays a set of "Shader Texture" images (abstract, stylized green and pink patterns) and "Base Image" examples (photorealistic or stylized close-ups of leaves, grass, and water droplets).] ### 00:13:48 # Animating the Orb in real time The shader combines a generated base image with live audio input to render the Orb in the browser. WebGL powers the interactive experience. OpenGL supports server-side rendering. ### Shader Playground Layers & Debug tools [Abstract gradient image with pink, purple, and orange hues] [Circular abstract gradient image with pink, purple, and orange hues] [Screenshot of a dark-themed debug console or code editor showing parameters and values] ### 00:14:15 # Animating the Orb in real time The shader combines a generated base image with live audio input to render the Orb in the browser. WebGL powers the interactive experience. OpenGL supports server-side rendering. ### Shader Playground Layers & Debug tools [Abstract image with swirling pink, purple, and orange colors] [Circular image of an "Orb" with swirling pink, purple, and orange colors] [Screenshot of a dark-themed debug console or shader editor showing parameters] ### 00:14:52 # Inside the shader - Audio energy disturbs a simulated flow field. **Fluid simulation** - Adds idle motion between spoken phrases. **Simplex noise** - Layers noise at different scales to add surface detail. **Fractal Brownian Motion (FBM)** - Shapes the boundary and creates a sense of depth. **SDF masking** - Adds a sharper visual response to audio. **Feathered sound wave** - Adds a subtle surface texture. **Film grain** [Circular image showing swirling green, purple, and yellow hues, representing fluid simulation] [Circular image showing a sphere split into red and green halves with a soft gradient, representing simplex noise] [Circular image showing a mottled, textured sphere in red and green, representing fractal Brownian motion] [Circular image showing a sphere with a gradient of blue, purple, and pink, representing SDF masking] [Circular image showing a black sphere with a white, crescent-shaped wave, representing a feathered sound wave] [Circular image showing a grey sphere with a subtle grainy texture, representing film grain] ### 00:15:19 - Layers noise at different scales to add surface detail. - Fractal Brownian Motio... - Adds a sha... l response to audio. - Feathered sound wav... ### 00:15:54 We built an across a la Seeing the variation, [Screenshot of a user interface showing a scrollable list of voice profiles or characters, each with a circular image, a name, and ### 00:16:19 # Making the system affordable at scale Generating a new image for every voice would be too slow and expensive. We used the voice analysis to group similar characteristics and pre-generate a catalogue of base images. - 3.6M$ savings in inference cost - 10-15x faster - Grouping voices by characteristics - 5x5 Matrix Full Matrix - 2x5 Light Matrix - < 3s per voice ### 00:16:50 - ...ould be too slow and - ...ive. - ...d the voi...ysis to gro... - ...character...and - ...erate a ca...ue of base ### 00:17:20 ### Making the system affordable at scale - Generating a new image for every voice would be too slow and expensive. - We used the voice analysis to group similar characteristics and pre-generate a catalogue of base images. - **3.6M$ savings** in inference cost - **10-15x faster** - **< 3s per voice** - **Grouping voices by characteristics** - 5x5 Matrix Full Matrix - 2x5 Light Matrix [Abstract gradient image with green and purple hues] ### 00:17:50 # End-to-end pipeline From voice sample to visual assets ## Voice analysis - **AUDIO ANALYSIS** - ERB PITCH: 5.69 ### 00:18:22 # End-to-end pipeline ## From voice sample to visual assets ### Voice analysis - **AUDIO ANALYSIS** - ERB PITCH: 5.69 - ### 00:18:53 ## Quality & Deployment review - Best: 2 hrs - Good: 30 mins - Better: 1 hr 29 minutes provided Continue to add recordings for a better ### 00:19:23 - One service across products - Shared API ### Give your text a voice - 10,000+ voices ### Cross Platform Deployment - Core app - Marketing ### 00:19:54 # IIElevenLabs ### Cross Platform Deployment - Core app, - Marketing website, - iOS and Android, - ElevenMusic, - Brand tools, and motion design. ### ### 00:20:18 ### Screenshot of ElevenLabs website homepage Bringing technology to life [Screenshot of the ElevenLabs website homepage] ### 00:20:48 # Voices [Screenshot of the ElevenLabs web application showing a list of user-created voices] ### 00:21:20 # Beyond the interface - **Paris CDG** - La santé n'attend pas. - Health can't wait. - **London** - Millions of journeys. - One voice that feels personal. [Two images of digital displays in public spaces. The left shows a vertical screen in an airport setting, displaying an ad for "alan" with the text "La santé n'attend pas. Health can't wait." The right shows a large horizontal billboard on a building in London, displaying an ad for "trainline" with the text "Millions of journeys. One voice that feels personal." Both ads feature a similar abstract, glowing spherical graphic.] ### 00:21:49 ### Advertisement for 11Eleven Agents and ANAP 750 auto-écoles. Zéro appel manqué. [Image of two identical advertisements displayed in a subway ### 00:22:46 # Patterns revealed what single ### 00:23:47 I still care about **every single output** Now I design **the conditions** that produce it [Abstract background with streaks of red, orange, and purple] ### 00:24:18 - Mistral - Singfuse by ClickHouse - NVIDIA - AI Engineer (Paris) [NVIDIA logo] ### 00:24:51 - Mistral - langfuse by ClickHouse - NVIDIA - AI Engineer PARIS - Notion - sonar - minty - desco [Grid of company logos] ### 00:25:23 # How a **Transformers** model loads & runs in **vLLM** ## Transformers modeling backend Follow one model — **Qwen/Qwen3-VL-30B- ### 00:25:55 CONTEXT - THE ENGINE ## What is vLLM? An open-source engine for LLM inference and serving – started at UC Berkeley's Sky Computing Lab, now a Py ### 00:26:21 # What is vLLM? An open-source engine for LLM inference and serving – started at UC Berkeley's Sky Computing Lab, now a PyTorch Foundation project. Give it a model and it serves it as fast as possible: 1. **P**aged**A**ttention * `KV cache in fixed-size blocks` The KV cache is allocated in blocks, like pages of virtual memory. Instead of one contiguous slab per request – almost no waste, more sequences per GPU, and prefix caching falls out for free. 2. **C**ontinuous **b**atching * `scheduler: chunked prefill` Requests join and leave the running batch at every step instead of waiting for the slowest one to finish, with long prefills chunked and mixed in alongside decodes. 3. **F**ast **k**ernels * `attention: fused MoE - quantized GEMMs` FlashAttention / FlashInfer, fused-MoE and FP8 / INT4 matmul kernels, with `torch.compile` and CUDA graphs stripping out Python and kernel-launch overhead. 4. **S**cale out * `TP - PP - DP - EP - SP - CP` Tensor, pipeline, data, expert, sequence and context (prefill / decode) parallelism. The Qwen3-VL run in this talk uses the first four, across 8 GPUs. 5. **S**erve anywhere * `vllm serve - OpenAI-compatible API` One command starts an OpenAI-compatible server, on NVIDIA, AMD, TPU, Intel, CPU and more through hardware plugins. ### 00:26:49 CONTEXT • THE BACKEND ## A **Transformers** model, run by **vLLM**. Instead of a hand-written port, vLLM loads the 🤯 model class itself and swaps its hot modules for vLLM's own. It lives in `vllm/model_executor/models/transformers/`. - 🤯 **The model definition** the `modeling_*.py` `nn.Module` classes and their forward, unchanged - 🤯 **Config & weights** straight from the Hub checkpoint — in-library or the repo's own `auto_map` code ``` --model-impl auto (default): native vLLM class if one exists, else this backend --model-impl transformers: always this backend ``` ### 00:27:21 ...them all in sync forever. The checkpoint that la... Hub serves on vLLM same day. it costs nothing to k it: after the sw ### 00:27:50 ### BEFORE THE MECHANICS - THE MOTIVATION # Why you'd reach for the Transformers modeling backend. vLLM's registry has hundreds of hand-written model implementations. This backend is for when yours isn't one of them — or when you'd rather it never had to be. One 😴 definition, run on vLLM's kernels. 1. **Ship the model once** You already **train + release in Transformers format**. A lab that trains and distributes using Transformers shouldn't have to write a second/third/fourth implementation, open a vLLM/SGLang/etc PR, and keep them all in sync forever. The checkpoint that lands on the Hub serves on vLLM the same day. 2. **One definition — trainer and rollouts** **post-training** · **RL loops** GRPO-style stacks run Transformers for the trainer and vLLM for the rollouts. Using this backend, both sides execute the same module code. So the policy you sample from is the policy you're training — there is no second implementation to drift. And it costs nothing to pick it: after the swap every hot ### 00:28:22 ```bash $ vllm serve Qwen/Qwen1-5-30B-AWQ-Instruct-FP8 --model-impl transformers \ --tensor-parallel-size 2 --pipeline-parallel-size 2 \ --data-parallel-size 2 --enable-export-parallel \ --compilation-config.compile_mm_encoder True ``` ### The CLI hands the model tag to the engine. `main()` in `entrypoints/cli/main.py` dispatches the `serve` subcommand. The positional `model_tag` becomes `args.model` and `--model-impl transformers` sets `model_impl`. The parallel sizes and `compile_mm_encoder` fix the 8-GPU layout the rest of this walkthrough assumes -- **EP** is TPxDP, so 4. The engine then builds a `ModelConfig`, and the first thing that does is read the checkpoint. ```python vllm/entrypoints/cli/serve.py ServeSubcommand.cmd def cmd(args: argparse.Namespace) -> None: if hasattr(args, "model_tag") and args.model_tag is not None: args.model = args.model_tag # the model tag ... ### 00:28:53 ```bash $ vllm serve Qwen/Qwen1-VL-30B-A1R-Instruct-FP8 --model-impl transformers \ --tensor-parallel-size 2 --pipeline-parallel-size 2 \ --data-parallel-size 2 --enable-export-parallel \ --compilation-config.compile_mm_encoder True ``` ### The CLI hands the model tag to the engine. `main()` in `entrypoints/cli/main.py` dispatches the `serve` subcommand. The positional `model_tag` becomes `args.model` and `--model-impl transformers` sets `model_impl`. The parallel sizes and `compile_mm_encoder` fix the 8-GPU layout the rest of this walkthrough assumes — **EP** is `TPxDP`, so 4. The engine then builds a `ModelConfig`, and the first thing that does is read the checkpoint. ```python vllm/entrypoints/cli/serve.py ServeSubcommand.cmd def cmd(args: argparse.Namespace) -> None: if hasattr(args, "model_tag") and args.model_tag is not None: args.model = args.model_tag # the model tag ... uvloop.run_server(args) # -> ModelConfig ``` ### 00:29:22 ```bash $ vllm serve Owen/Owen3-VL-30B-A3B-Instruct-FPB --model-impl transformers \ --tensor-parallel-size 2 --pipeline-parallel-size 2 \ --data-parallel-size 2 --enable-expert-parallel \ --compilation-config.compile_mm_encoder True ``` ## The CLI hands the model tag to the engine. `main()` in `entrypoints/cli/main.py` dispatches the `serve` subcommand. The positional `model_tag` becomes `args.model` and `--model-impl transformers` sets `model_impl`. The parallel sizes and `compile_mm_encoder` fix the 8-GPU layout the rest of this walkthrough assumes — `EP` is `TPxDP`, so 4. The engine then builds a `ModelConfig`, and the first thing that does is read the checkpoint. ```python ... vllm/entrypoints/cli/serve.py ServeSubcommand.cmd def cmd(args: argparse.Namespace) -> None: if hasattr(args, "model_tag") and args.model_tag is not None: args.model = args.model_tag # the model tag ... uvloop.run_server(args) # <-- ModelConfig ``` ### 00:29:55 # A vLLM Qwen3VLMoeforConditionalGeneration exists — we ask for the backend anyway. - `--model-impl transformers` forces the backend (with auto, v