# How a Transformers Model Loads in vLLM — Harry Mellor — session 2026-09-24T08:30:00.000Z → 2026-09-24T09:00:00.000Z _36 transcript lines · 36 slides_ ## Transcript Model. And so we choose the transformers multimodal MOE for course LLM class. And what are these classes made of? Because there were quite a lot of them, and you'd think that if we didn't do anything clever, we would end up with lots of duplicated code and lots of drift, and it would be really hard to maintain. But all of these are made of mix ins which enable each individual capability. So for this one, we first start with the MOE mix in, which provides all of the necessary Protocols for expert parallel and expert parallel load balancing. Then we have the multimodal mix in, which handles all of the vision encoding, the encoding the vision encoding caching, MROPE if your model has it, which this one does, and then merging the vision embeddings into the text embeddings before passing it over to transformers. Then the causal mix in, which is for the LM head and the logics computation. And then finally, the base class, which does most of the heavy lifting, and all of the mix ins actually now inherit this, and it ends up at the bottom of the inheritance stack here. And that handles the weight loading, the weight mapping, all of the modifications that enable this to run quickly, just as fast as a VLLM model would. And then there is a a whole host of supports this, supports that, protocols for things like quantization, LoRa, pipeline parallel. Eagle and eagle three. So now we're going to look at how the transformers class is instantiated and what we do to it, during instantiation. So it all looks a bit faint now, but that's because it has not yet been instantiated. First, we actually before doing any of that, we register two attention shins with transformers attention registry, and that is the VLLM attention and VLLM LA attention forward methods. Then we patch the config so that when the forward pass of the transformers model happens, one of those shims, depending on which is correct for your model, is what is actually selected rather than, flash attention, flex attention, or whichever other options the model, claims that it can use. Then we decorate, the language model always for. Compile from VLLM's perspective. And then because of that command line argument I mentioned earlier, we also decorate the vision tower, which tells VLLM that you can compile both of these as whole graphs independently of each other before TorchCompile gets a chance to wing it and figure out on its own. And now we instantiate the model using the AutoModelFromConfig method on the meta device. And for AutoModel is slightly different to, AutoModel for Corswell m, which might be a class you're more familiar with if you use the transformers ecosystem. And the AutoModel just loads what transformers refers to as the base class, which is just the language backbone and the vision tower but without the language modeling head. I might have to step back to the laptop. This appears to have stopped working. Then we create the HF to VLLM mapper, which is how we specify how checkpoint weights might be translated into how they now look in the Python class that gets created. Transformers has a few of these, built in if they have moved from an old style naive mixture of experts to a fused mixture. Of expert style. So at this stage, this is a pretty minimal mapper, but as we move through this process, we'll be contributing to it as we modify the model. So after that, we then delete all of the layers that the current pipeline stage is not using. And in this example, we're on rank zero, so we delete the second half of the layer stack and the final layer norm. And now we begin recursive replace, which is The bulk of the work happens to bring the performance and access to all the features that VLLM has. For MOE models, it happens in two passes because the MOE is a little bit special, and there are two different ways which we can do it. For this model, it uses the internal routing method, which means that we, by using Torch FX, detect the structure of the sparse MOE block. And if we determine that it is entirely represent By VLMs MOE runner, then we replace it wholesale. So we replace the experts with this MOE runner, and that contains the routed experts. And then the router gets replaced with a gate linear, and that gets passed into the MOE runner, and that handles everything. And so that lets us do, things like use monolithic kernels for routing and experts. If there were shared experts in this model, it would allow us to do asynchronous. Shared expert computation, and all of that is managed by the MOE runner. And then once that is done, we go to the second pass, which is the base class doing a depth first search through all of the children of the base model. So first, we walk through the vision model, replacing all of the layers that we know how. To its features. So all of VLLM's parallelization is built into its layers. So by using VLLM's linear layers and VLLM's, expert's layers, That's how we gain access to Tensor parallel, expert parallel, etcetera. Then we do the vocab parallel embedding swap, by surgically replacing just the Torch NN embeddings module. So if you have a model like Gemma where you do some scaling during your token embedding, that is all still preserved, and we don't have to special case it on the VLLM side. Now the next fuser is used, which is the QKB fuser, which, again, uses FX to detect the QKV and output projections, and swap in BeadleM's QKV, parallel linear, and a corresponding row parallel linear, using AST. So so this is all it's using Torch FX to detect what we need to swap, but then it's using Python's AST module to actually make the modifications. So we are yet to torch compile anything, and all of that is Still left up to later stages in VLLM. Then similarly, we replace the RMS norm, which is deceptively difficult for how simple, it is, But we do it because VLLM uses some global Torch compile passes to detect where it can fuse layers, and it does it by detecting its own modules. And so there are some, RMS norm. Pass it or some compilation passes that the RMS norm is involved in that will only be detected if we're using VLLM's class. Then we create VLLM's attention instances. And as before, this gives us access to the rest of VLLM. And the key thing that this gives us access to is VLLM's kv cache. And so we don't need to use any kv caching that has been implemented on transformers, and VLLM gets to do all of the clever stuff that it can do with the kv cache. Cache, like offloading, disaggregated prefill between nodes, kv cache reuse, so all of that fun stuff. Then anything that we haven't touched yet, we make sure that it is on the device. So the only thing that changed there was those two norms in the vision model. And then lastly, we just make sure that anything that transformers insist should be in FP 32 stays in FP 32 even if the rest of the model is FP eight. So we've built the model, and now a request has been received. We have had it scheduled through VLLM, and we've reached the base class's forward method. And this is the first hop back into transformers. So we simply call the forward method on the original quen3vlmoemodel class and pass in the input IDs or the input embeds. This case, it's the inputs embeds because it's a multimodal model, and VLLM pre embeds the vision part, of the prompt and inserts it into the text embeddings and passes that all in one go rather than relying on the model itself to to do that. We explicitly disable transformers' kv cache because VLLM owns it through those attention modules. And then we just forward through the position IDs with a Modification for emerald models, which happens in the multimodal mixing for this particular type of model. And so while we're going through the forward pass of the transformers model, there are two scenes where we return back to VLLM, in kind of performance critical ways. And so the first and most interesting one is the attention module. So on the transformers side, we're starting again. In the the transformers native class. It goes through its registry to choose which attention method to use, which we preregistered earlier as the VLLM, attention shim, and then selected it by modifying the config. And then we enter that and transition back to VLLM, where VLLM does some minor shape manipulation, gets the attention kernel to use, which is any attention kernel supported by your Device for this particular model in VLLM. It gets the kv cache using, the attention context, and then it actually calls the attention and passes the result back to transformers to continue with the rest of the model. And the second is the MOE through internal routing. So as I mentioned, the MOE has a special case where if we're able to detect an entirely representable sparse MOE block, then we can Replace it entirely. And actually, the forward method that we use is just experts forward, and everything happens inside that MOE runner class. So we go straight into the VLM side. We gate the hidden states. We get the top k IDs and logits, perform the rooted experts, the computation, and pass the results back to transformers. And after this Point, the transformers model is done. It has provided all of the information about what the graph of the model is and all the computation for well, and all the journey through the computation, rather, because as we saw, the modules are all VLLM modules. And then after that, the logic computation and LM head evaluation is all done by VLLM, and the sampling is also handled by VLLM. So there are a few Features that we haven't used with this model that I want to talk about. The first is external MOE routing. So if your MOE block isn't the same as something that we've anticipated, That doesn't mean that it won't work. It just means that we will only replace the experts themselves. So that gives you the crucial performance because that's where most of the work is done, but it gives you the flexibility to do whatever you might come up with in the The routing or anything around it. And there's also shared experts, which I alluded to earlier. If they had existed and they were detected in the internal routing, they would also have been included in the MOE runner. And depending on your configuration of BLLM, it would be computed in a way that is more efficient than could be done with external routing. Then there's LoRa and Eagle and Eagle three, which we Didn't touch on because these are kind of determined by the checkpoint you use and then the requests that you pass. Then there's a a whole load of different attention flavors. So there's encoder only attention, which would be used by models like BERT or sliding window attention that you might find in Gemma models. And for encoder, we support encoder only models, not no encoder decoder models. But for sliding window Attention. It can be interleaved, similar to how they started doing it with Gemma where you would alternate between full and sliding attention, and then also MLA attention, which comes through a separate, fuser. And that actually gives us the kv cache compression benefits that we didn't used to get when we used the we just used to pad to full attention when you loaded a a MLA attention model. And then finally, pooling and classification tasks, which would have been automatically used had we used a, a pooling model. Or we could have manually selected it, and DLLM would have created a classification head using the, the LM head, or it could have given you back the the raw embeddings out the end of the model if that is what you were after. And so to recap, we Looked into the config, determined which Hugging Face architecture it was, and then which transformers modeling backend class we should use. We loaded it into this generic class in VLLM, and we manipulated it such that all of the advanced features of VLLM became accessible, and all of the performance that comes with them came with them. Then we went through the forward pass of the base model with two important seams, We go back into VLLM and do some nontrivial, work, and then we pass back to VLLM to do the logic computation and the sampling. And that was my talk. And the slides are available on the QR code there, if anyone wants to look at them again later. Thank you. Sorry. Alright. That's what happens. Alright. I think we're on a break now. You so much, everybody. So I think we were gonna take a little break, and we're back in half an hour, I believe. Alright. Thank you so much, ## Slides ### 00:00:23 # A vLLM **Qwen3VLMoEForConditionalGeneration** exists — we ask for the backend anyway. ```python # --model-impl transformers forces the backend (with auto ### 00:00:48 ### C3 LINEARIZATION # One class, a stack of jobs. ```python class TransformersMultiModalMoEForCausalLM(MoEMixin, MultiModalMixin, CausalMixin) ``` – every mixin subclasses Base, so C3 folds it in exactly once, at the foot of the tower. Each mixin owns one concern and cooperatively `super()`-chains, skipping the marker interfaces so their `__init__` never runs twice. ```python super(MixtureOfExperts, self) = super(SupportsMoEPR, self) = \ super(VLLMModelForTextGeneration, self) = Base.__init__ ``` - `does real work` - `marker interface` The class Python instantiates: `TransformersMultiModalMoEForCausalLM` ### 00:01:22 ## C3 LINEARIZATION ### One class, a stack of jobs. ```python class TransformersMultiModalMoEForCausalLM(MoEMixin, MultiModal ### 00:01:50 ### C3 LINEARIZATION # One class, a stack of jobs. `class TransformersMultiModalMoEForCausalLM(MoEMixin, MultiModalMixin, Causal ### 00:02:26 # vLLM BASE.PY `__INIT__` 8 GPUS: PP2 TP2 DP2 EP4 FP8 (TEXT TOWER ONLY) COMPILE_MM_ENCO ### 00:02:50 # Load the 🤯 graph on meta, then **split, rewire & quantize it.** ```python vLLM BASE_PV __INIT__ 8 GPUS: PP2 ### 00:03:23 # Load the 🤯 graph on meta, then split, rewire & quantize it. ```python ALL_ATTENTION_FUNCTIONS.register("vllm", vll ### 00:03:52 # Load the graph on meta, then split, rewire & quantize it. ```python vLLM BASE.PY __INIT__ 8 GPUS: PP2 TP2 ### 00:04:19 # Load the 🤯 graph on meta, then split, rewire & quantize it. ```python ALL_ATTENTION_FUNCTIONS.register("vllm", vllm_ ### 00:04:54 # Load the 🤯 graph on meta, then **split**, **rewrite** & **quantize** it. `vLLM` `BASE.PY __INIT__ ### 00:05:18 # Load the 🤯 graph on meta, then **split, rewire & quantize it.** ```python BASE.PY _INIT_ _GPUS: PP2 TP1 DP2 EP4 ### 00:05:54 # Load the graph on meta, then split, rewire & quantize it. ```python vLLM BASE.PY __INIT__ B GPUS: PP2 TP2 DP ### 00:06:23 ```python pipeline_parallel() . PP = 2 # Base recursive_replace() # MoEMixin -> Base ``` ### 00:07:22 # Load the graph on meta, then split, rewire & quantize it. ```python vLLM BASE.PY __INIT__ ALL_ATTENTION_FUNCTIONS.register ### 00:07:50 ## recursive_replace() ### 00:08:23 # Load the 🤯 graph on meta, then **split, rewire & quantize it.** ```python vLLM BASE.PY __INIT__ 🤯 GPUS: ### 00:08:50 # Load the 🤯 graph on meta, then **split, rewire & quantize it.** ```python vLLM BASE.PY __INIT__.PY 8 GPUS: PP2 ### 00:09:19 # Load the 🤯 graph on meta, then **split, rewire & quantize it.** ```python vLLM BASE.PY __INIT__ 8 GPUS: PP ### 00:09:55 ### MULTIMODALMIXIN.FORWARD (M-ROPE RESHAPE) -> BASE.FORWARD -> OWEN3VLMOEMODEL.FORWARD # The scheduler hands ### 00:10:23 ## MULTIMODALMIXIN.FORWARD (M-ROPE RESHAPE) -> BASE.FORWARD -> GWEN3VLMOEMODEL.FORWARD ### The scheduler hands vLLM ### 00:10:55 ```python get_attention_context() # pulls attn_metadata + kv_cache impl.forward(...) # FlashAttn infer kernel ``` **hidden states** port: the shims are registered into 🐒's ### 00:11:22 # _ATTN_IMPLEMENTATION = "VLLM" • THE ELEGANT SEAM ## defines attention — **vLLM runs it.** Setting `_attn_implementation="v ### 00:11:54 # _ATTN_IMPLEMENTATION = "VLLM" THE ELEGANT SEAM ## defines attention — **vLLM runs it.** Setting `_attn_implementation=" ### 00:12:24 # defines the block — **vLLM replaces it whole.** MoEBlockFuser matches Qwen3-VL's text block, and `rewrite_forward` swaps its **forward** ### 00:13:21 ### What Qwen3-VL-30B-A3B-FP8 doesn't use. The same **Base** + mixins wrap every architecture. For Qwen3-VL-30B-A3B-FP8, vision, M-RoPE, FP8 quant and full 8-GPU parallelism (PP2:TP2:DP2:EP4) are all used — but plenty is checked, and skipped: - External MoE routing - `TransformersMoERunner + custom op` - The fallback when `MoEBlockFuser match()` can't trace a block's router as `topk(score(linear(x)))` — Transformers keeps the gate and the chosen `topk_ids` are smuggled into VLLM's `routedExperts` via a `custom_routing_function`. ### 00:13:49 ### What Qwen3-VL-30B-A3B-FP8 doesn't use. The same **Base** + mixins wrap every architecture. For Qwen3-VL-30B-A3B-FP8, vision, M-RoPE, FP8 quant and full 8-GPU parallelism (PP2.TP2.DP2.EP4) are all used — but plenty is checked, and skipped: - **External MoE routing** - `TransformerMoERunner` + `custom op` - The fallback when `MoEBlockFuser.match()` can't trace a block's router as `topk(score(linear(x)))` — Transformers keeps the gate and the chosen `topk_ids` are smuggled into vLLM's `routedExperts` via a `custom_routing_function`. - **Shared experts** - `SharedExpertMLP` - Qwen3-VL-MoE has none. For models such as DeepSeek/GLM, `MoEBlockFuser.match()` finds them and passes them to `MoERunner`. ### 00:14:23 # What Qwen3-VL-30B-A3B-FP8 doesn't use. The same **Base** + mixins wrap every architecture. For Qwen3-VL-30B-A3B-FP8, vision, M-RoPE, FP8 quant and full 8-GPU parallelism (PP2.TP2.DP2.EP4) are all used — but plenty is checked, and skipped: - External MoE routing - `TransformerMoERunner + custom_op` - The fallback when `MoEBlockFuser.match()` can't trace a block's router as `topk(score(linear(x)))` — Transformers keeps the gate and the chosen `topk_ids` are smuggled into VLLM's `RoutedExperts` via a `custom_routing_function`. - Shared experts - `SharedExpertMLP` - Qwen3-VL-MoE has none. For models such as DeepSeek/GLM, `MoEBlockFuser.match()` finds them and passes them to `MoERunner`. - Encoder & sliding-window attn - `EncoderOnlyAttention` `+ sliding_window` - `EncoderOnlyAttention` is only used for encoder-only models like BERT, this multimodal decoder takes the DECODER path. All attention layers in this model are full, so no per-layer sliding attention either. - MLA (latent attention) - `MLAFuser` `MLAttention` "The `vita_ata` skip ### 00:14:50 # What Qwen3-VL-30B-A3B-FP8 doesn't use. The same **Base** + mixins wrap every architecture. For Qwen3-VL-30B-A3B-FP8, vision, M-RoPE, FP8 quant and full 8-GPU parallelism (PP2.TP2.DP2.EP4) are all used — but plenty is checked, and skipped: - External MoE routing - `TransformerMoERunner` + `custom op` The fallback when `MoEBlockFuser.match()` can't trace a block's router as `topk(scoreLinear(x))` — Transformers keeps the gate and the chosen `topk_ids` are smuggled into VLLM's `routedExperts` via a `custom_routing_function`. - Encoder & sliding-window attn - `EncoderOnlyAttention` + `sliding_window` `EncoderOnlyAttention` is only used for encoder-only models like BERT, this multimodal decoder takes the DECODER path. All attention layers in this model are full, so no per-layer sliding attention either. - ### 00:15:22 # What Qwen3-VL-30B-A3B-FP8 doesn't use. The same **Base** + mixins wrap every architecture. For Qwen3-VL-30B-A3B-FP8, vision, M-RoPE, FP8 quant and full 8-GPU parallelism (PP2·TP2·DP2·EP4) are all used — but plenty is checked, and skipped: - **External MoE routing** `TransformersMoERunner` + `custom op` The fallback when `MoEBlockFuser.match()` can't trace a block's router as `topk(score(linear(x)))` — Transformers keeps the gate and the chosen `topk_ids` are smuggled into vLLM's `RoutedExperts` via a `custom_routing_function`. - **Shared experts** `SharedExpertMLP` Qwen3-VL-MoE ### 00:15:51 # Transformers defines the model. vLLM owns the machine. ## RECAP The model graph and RoPE live in 🪖 — both towers, text and vision. ### 00:16:20 - One model. One config flag. - Every new Transformers architecture - AI Engineer Paris 2023 ### 00:17:20 - Mistral - AI Engineer PARIS - langfuse by ClickHouse - Sonar - NVIDIA - cominty [Logos of Mistral, AI Engineer, Langfuse, Sonar, NVIDIA, and Cominty] ### 00:17:48 # AI Engineer Paris Organized by: Mistral ## Platinum - NVIDIA ## Gold - Cockroach Labs - FACTORY - SENTRY - Sonar ## Silver - CLOUDFLARE - Daytona - langfuse by ClickHouse - Modal - neo4j - Qdrant ## Bronze - CodeRabbit - cominty - descope - Latitude - MAKORA - FMO - stripe - ZenMLabs ## Supporting partners - Backblaze - Notion - SIERRA ### 00:25:15 # AI Engineer Paris Organized by Mistral ## Platinum - NVIDIA ## Gold - Cockroach Labs - FACTORY - SENTRY - Sonar ## Silver - CLOUDFLARE - Daytona - langfuse by ClickHouse - Modal - neo4j - Qdrant ## Bronze - CodeRabbit - cominty - descope - Latitude - MAKORA - FMO - stripe - ZenMLlabs ## Supporting partners - Backblaze - Notion - SIERRA ### 00:26:47 # AI Engineer Paris Organized by - Mistral ## Platinum - NVIDIA ## Gold - Cockroach Labs - FACTORY - SENTRY - Sonar ## Silver - CLOUDFLARE - Daytona - langfuse by ClickHouse - Modal - neo4j - Qdrant ## Bronze - CodeRabbit - cominty - descope - Latitude - MAKORA - FMO - stripe - ZenMLlabs ## Supporting partners - Backblaze - Notion - SIERRA ### 00:27:16 # AI Engineer PARIS ## Organized by - Mistral ### Platinum - NVIDIA ### Gold - Cockroach Labs - FACTORY - SENTRY - Sonar ### Silver - CLOUDFLARE - Daytona - langfuse by ClickHouse - Modal - neo4j - Qdrant ### Bronze - CodeRabbit - cominty - descope - Latitude - MAKORA - FIMO - stripe - ZenMLlabs ### Supporting partners - Backblaze - Notion - SIERRA