# Beyond Generation: Fast, Structured AI Decisions with Jev at Glean — Chinmay Goyal — session 2026-11-11T22:30:00.000Z → 2026-11-11T22:43:00.000Z _30 transcript lines · 24 slides_ ## Transcript Thank you. Thank you. Hi, everyone. My name is Chinmay, and I'm a software engineer at Clean. I work on the product engineering side of Clean Assistant. Today, I'm going to give a talk on how Lean uses Jamf. The title of the talk is Beyond generation, fast and structured AI decisions with GenWeb Clean. If you look at it, most AI product steps are bounded decisions and not generation problems. So to give a couple of examples, user query intent classification, a ranker picking a suggestion out of n options, checking whether a citation is valid or not, routing to the correct model or expert, so on and so forth. Till now, we have Been paying the full LMM cost and latency to get an answer to all these bounded problems. Jev at Glean fills the gap between general LMMs and fine tuned classifiers. Classification itself isn't a new problem. Like, people have been trying to solve it, since forever. We can have multiple ways to solve it. Some other things that we have tried out include using a general purpose LLM, using an open model, something like. Fine tuning that open model, or using JEV. Now, all these approaches and all these, different ways have their pros and cons. With a general purpose LLM, we incur a lot of cost and latency. While with something like JEV, we get no generated text output. Another advantage that we have been seeing with Jiv is that as the taxonomy of the decision space changes, we don't have to fine tune these models again and again. We can just change our prompt. And boom, it works out of the box. Now I'll walk you through how we have integrated Gen or these classification, how we have used JavedGLIN to solve these classification problem statements. I'll go into some of these problems in great detail. Before we go into these, individual problems, I wanted to touch upon how we evaluate these changes. Like, we have a huge Customer base having millions of users. So we don't want any of the change that we make to regress our product. We have a multi phase approach, a rollout approach to any change that we are making. This includes having a golden eval set for almost every possible thing. Like, any place where we have generative AI, any place where we have any form of intelligence involved, we have a golden eval set for that space. We soak all of these changes for a significant amount of time for our internal use case. Like Have around 1,500 people at Glean right now. We soak all of these changes within our Glean instance for, like, a couple of weeks. We get active feedback. We have a very active feedback loop from our internal users. We work upon that. And then we go for an AB testing with our customers. The first problem that we tried out with our decision model was query classification. Query classification is when we want to know the nature of user queries. Realize what kind of what kind of jobs are people using clean for. What kind of... Say, for example, if people are using more for sales like queries, we focus more on that. If people are using more for engineering type use cases, we focus more on that. We tried out three different approaches over here. And this is data from our test bed. We used an open source layer model. Then we tried out JEV. And then we fine tuned the layer model on our own compute resources. JEV is the best no training option in our opinion. It nearly doubles the output quality of a base layer model and is able to perform at a 90% quality in comparison to a fine tuned model, a fine tuned layer. With JEV, we don't have to go through the very tedious process of data collection, data cleaning, incurring the cost for compute, like training the model, fine tuning the model again and again as the taxonomy changes. And at the same time, we don't have to worry about hosting the model or, incurring the inference cost. The next problem we went to is expert routing. This is a pretty common problem statement in the industry right now. Like, hey, when the user query comes in, what expert do we route it to? Like, how do we ensure that the user, user... The response to the user is of the best possible quality? Glean receives... The way we have been solving this is, Glean receives the user query and then decides what The best expert to answer the query. Till now, we have been using a general purpose LLM to route to the correct expert. We set up a test bed. For JEV, we set up a test bed with n experts. And our experiments found a significantly high precision when using JEV and that too at an almost 8x reduction in latency. We had to change our implementation or our infrastructure significantly to integrate Jamf into the specific stage of our product. But the reduction in latency It justifies that inherent complexity in the system. We at Glean have been trying to build systems in the most... In the simplest possible manners. This is one of the rare pieces where we believe complexity is worth it. Next, I'll walk you through the ranter problem statement that I talked about. When we receive a user query, we also want to generate the best follow-up suggestions for the user. Like, this helps us in getting discoverability for our product. This helps us in ensuring that the user The best possible experience. You also want to recommend or say preempt what the next user query might be. Again, a pretty common problem statement that the entire industry has been trying to solve for. Internally, we call this system q, like queue as in the literal queue. The way this works, we have like 10 or 15 different sources or 10 or 15 different ways, 10 or 15 different sources that are generating suggestions independently of each other. Once those suggestion... These suggestions are generated by natural language LLMs and it's like a proper natural language query. Once these suggestions are in, we have a ranking step. So the ranker takes in all these suggestions, takes in the u... The first user query, takes in the output, that Glean gave, and then all these suggestions in that context. The ranker is then asked to decide like, hey, you have these suggestions. What do you think is the best one? A pretty interesting bottleneck that we had over here is we want the response to be snap. I cannot... We cannot have a state where we give the textual response to the user and then wait five seconds for generating the follow-up suggestion. We have to be snappy. That's why we have time limits on these sessions. So right now, the step had a time limit of one one point five to two seconds. Jeff, no doubt, gave us a better quality than... Better quality and far better latency than what we had with the general purpose LLM. But two things stood out over there. First, we were able To give a suggestion within our time bounds in 99.2% cases. With the general purpose LMM, this was almost 90%. So this ensures that we have user engagement in far more user queries now. The second thing is at the clean, whenever we use these LLMs, we have guardrails. Guardrails in the sense that, hey, LLM might not be good at generating a response or LLM might be making some specific kind of mistakes. So we have Checks and balances for all those possible mistakes that we see. We keep a log of, like, we keep a log of all these checks that are incurred at any point of time. With a general purpose LMM, we were able to generate a valid suggestion in almost 90% of the use... 90% of the cases. We still had some places where LLM was either generating a lang... Generating a response in a different language. The output did not, meet certain other criterias that we had had. ## Slides ### 00:00:09 # Thank you - early believers - reliability realists - software researchers, dreamers, contrarians typesafe.ai [Abstract geometric logo] ### 00:00:40 # AI Engineer ## Code Summit ### 00:01:12 # AI Engineer Europe 2027 ## ExCeL London ### February 17-19, 2027 Europe's flagship technical AI conference returns to ### 00:01:36 [Abstract pattern of dark grey dots on a black background, creating a subtle, undulating sphere-like shape in the center] ### 00:02:06 # Code Summit ## AI Engineer ### 00:02:38 # Code Summit ## AI Engineer LIVE FROM SF//CA [Map outline of San Francisco in the background] ### 00:03:08 [Dark background with a subtle, wavy grid of grey dots] ### 00:03:38 # WESLSL < HC 37° 37' 08" N 125° ***% JV> 6 [Stylized map in the background with grid lines] ### 00:04:06 # AI Engineer Code Summit Presented by Google DeepMind // AVAILABLE ## LIVE TRANSLATION Scan the QR code to access real-time translation. [QR code linking to https://events.google.com/codesummit/live-translation] ### 00:05:12 # Beyond Generation: Fast, Structured AI Decisions with Jev at Glean Chinmay Goyal / Software Engineer glean [https://aie.sc/](https://aie.sc/) [https://ai.engineer/main](https://ai.engineer/main) [Two QR codes are displayed on the right side of the screen.] ### 00:05:36 # AI Engineer Code Summit ## Beyond Generation: Fast, Structured AI Decisions with Jev at Glean ### Chinmay Goyal / Software Engineer at Glean * user asking for: ### 00:06:08 # Beyond Generation: Fast, Structured AI Decisions with Jev at Glean ## Jev fills the gap between general LLMs and fine-tuned classifiers ### General-purpose LLM **WHY ### 00:06:36 ## Jev fills the gap between general LLMs and fine-tuned classifiers ### General-purpose LLM **WHY USE IT** - Zero-shot, flexible, can explain itself ### 00:07:08 # What we tested at Glean Five bounded decisions, each measured against an existing baseline - Query classification - Model routing - Suggestion ranking - Citation judging - Guardrails [Five distinct, rounded rectangular elements listing the decisions tested] ### 00:07:40 # We evaluate in stages ## before anything reaches users 1. **Agreement with production** * Query classification: 6,318 records vs LLM pipeline ### 00:08:08 # We evaluate in stages before anything reaches users 1. **Agreement with production** * Query classification: 6,318 records vs LLM pipeline labels 2. **Golden-set replay** * Routing: 751 labeled entries, precision and recall per route 3. **Repeatability runs** * Citations: 3 identical runs, verdict changes across 28 paragraphs 4. ### 00:08:39 ## Query classification: 90% of fine-tuned quality, zero training - **90%** of fine-tuned Laya's agreement (66.8% vs 7 ### 00:09:12 # Query classification: ## 90% of fine-tuned quality, zero training - **90%** of fine-tuned Laya's agreement (66.8 ### 00:09:35 # Routing to the right expert: more precise and **8.1x faster** **Precision by route, prompt-based LLM router vs Jev (751 golden ### 00:10:07 # Routing to the right expert: more precise and 8.1x faster ## Precision by route, prompt-based LLM router vs Jev (751 golden entries) ### 00:10:36 ## Routing to the right expert: more precise and **8.1x faster** ### Precision by route, prompt-based LLM router vs Jev (751 golden entries) ### 00:11:08 # Picking the suggestion: one Choice call made the ranker 7x faster After each answer, a ranker picks at most one follow-up (query, artifact, or skill), or ### 00:12:08 # Picking the suggestion: one Choice call made the ranker 7x faster After each answer, a ranker picks at most one follow-up (query, artifact, or skill ### 00:12:39 # Beyond Generation: Fast, Structured AI Decisions with Jev at Glean ## Picking the suggestion: one Choice call made the ranker 7x faster After each answer, a ranker