# What Your AI Doesn't Know: The latest trends in agent searches — Jeffrey Pinner — session 2026-11-12T19:45:00.000Z → 2026-11-12T20:05:00.000Z _33 transcript lines · 39 slides_ ## Transcript Alright. Hello, everyone. My name is Jeff Pinner. I'm the CTO of Exa. Thank you guys for having us here. And we're gonna talk a little bit today about things your agent doesn't know. So web search has been around now for about Thirty years. Long enough where, at this point, we're kind of used to it, and we don't really question the status quo or how it could be different with agents. Over the last eighteen months or so, search has picked up agents as sort of a new customer. They're searching on our behalf. And so at Ekso, we specifically build web search as a knowledge layer for agents. We've gone from about 12 and a half million requests a month two years ago to a little over 1.3 Billion this past October, so about a 100 x over two years. And it's not unique to Exa. If you talk... OpenAI just announced that ChatGPT handles about a fifth of Google search volumes, and agents in general make up just around half the volume of web traffic now. And so I think the natural question is, why do agents need search at all? So let's start with trying to answer this before we talk about how we do search for agents. So frontier Models can do a lot of tasks from parametric knowledge, but unfortunately for real world real world tasks, that simply isn't enough. Generally, frontier models, no matter how capable they are, are always out of date. Model launches take two to five months from training cutoff to launch, then there's generally a few months after that before the next frontier model is released. And so taken that together, you're about four to eight months out of date with what The latest knowledge base is. And in the meantime, the world doesn't stop changing just because a model has moved from training to inference. Libraries launch. APIs change. Prices change. New regulations appear. Court rulings happen. The world continues to go. And even if that weren't true, training data itself is actually just a small amount of knowledge that the model is trained on. The biggest models out there are trained on around 50,000,000,000,000 tokens. The web itself is around 3,100, and that includes the deep web behind things like login and apps. If you focus it down to just the web that is indexed, the thing that search engines can see, that's still about 510,000,000,000,000 tokens, which means even if the world didn't continue to evolve after the model cutoff happened, you would still only be indexing a very small amount of the knowledge that's Out there that model... That models have to go and find. And for the type of tasks we want models to do, we want agents to do, there is valuable work that happens that is based on outside knowledge, whether it's across coding, legal work, due diligence work. Generally, all these tasks require correct, complete knowledge across multiple sources. And so this is what we're trying to... The gap we're trying to close. Now, the first thing to do when we're trying to close any gap is to try to measure it. And so the first thing we do is create a benchmark called ATLAS, which tries to test how well harnesses can leverage real world knowledge. So we created the Atlas benchmark to try to test agents in these tasks. First, we require agents to find entities that match a condition. Then we require agents to enrich the knowledge with as much information as they can find. And finally, we score them against the golden answer key, and we only count rows as correct that correctly identify all aspects of what we're looking for. And so to succeed at Atlas, you need to be consistently right and not just have a decent hit rate for individual searches. So we're gonna use this as a benchmark for how we measure agent performance when it comes to knowledge retrieval. And the first thing that we find is that model memory falls off with recency. When we test agents on Atlas, we see that model success rates fall off sharply as soon as they enter... As soon as they encounter data that's not covered in the training set. The newer the fact is, the less the models know. And models require, or I should say, agents require data to try to achieve these tasks. And so the first thing that we can see is that memory alone, parametric memory isn't sufficient. So when we run agents with just parametric memory without web search, they score around 9% on Atlas, and so we use this to try to understand quantitatively how web search improves performance. So the next natural question is, why don't they just And the first reason is agents don't search like people. Generally, people type about three words into Google, and that's up from about two words six years ago. Agents, on the other hand, search with more or less full sentences. The mean search on x is about 12 words. 10% of queries on x are 17 words or higher. And these words add constraints, versioning information, dates, places, all different types Of conditions on the on the search that keyword ranking wasn't really designed to handle. And so we'll take a look at an example of that. So this is an actual query from Exa. It's a request for companies. It's a full sentence. It adds a bunch of constraints. And when you feed this into a keyword retrieval algorithm, what you really get is a handful of results that don't really answer what the query asked for. The search didn't understand the query. The result... The third one is specifically in The US, and the result Asked for data outside The US. And so we're gonna come back to this in a few minutes because the second thing I wanna focus on is the fact that for a lot of knowledge on the web, search just doesn't have access to it. Most of the valuable information isn't on the open Web. Reported news, people data, financial data, research, they're all paywalled off. And so search engines that crawl the open Internet don't ever see these page pages. And then last Lastly, even when knowledge is on the open web and your agents can access it, there's a third problem, which is efficiency. You know, if an agent makes 10 searches, reads five pages from that search, gets about 16 k of raw HTML from each page. That's about 860,000 tokens just in processing results. And if you remember from the first slide, we serve about 1,300,000,000 requests a month. So that's a lot of tokens that are just churning to Tracked information. And then finally, when you add keyword search and we look at what happens to our ATLAS benchmark, the score definitely improves. It's added recent knowledge, but we've gone up to about 28% because most of the rows still come back with incomplete data or just wrong values. And so the challenge when we look at search is how do we actually provide the knowledge to agents that they actually require? So the first thing, and I told you we'd come back to this query, is, at least at Exa, the first thing we do to try to tackle this task is to extract meaning out of information as opposed to just syntactical matches or keyword matches. And so we look through pages. We understand the intent of queries, and here, we give you results that specifically answer the intent with all of the conditionals attached. And so in order to do this, we Do what you would for a modern web crawler. We crawl the Internet. We do a semantic extraction. We embed it in the vector space. We index it. When your query comes in, we embed the query in the same vector space, and we find the nearest matches. And this lets us move from syntactical understanding and keyword matching to a semantic understanding of the query in the web. And we do this at a volume of up to a million pages a second, with Latency of about three minutes from crawl to index. Now for the second problem, we need to extend it just beyond the open web. And so here, we have to extend what's on the public web with both proprietary data that's partnered and knowledge graph data that we extract and represent. And this way, the agent can retrieve questions and answers that it needs These questions without having to know where to look for them, doing this behind a single API. And for the last challenge, In order to support the volume of queries that agents make, we can't simply point them to web pages. We have to extract the answers from the page and create a token dense representation for agents to process. It's not sufficient to say the answer to this query lives on this web page. You could take, for example, this question of who won the Nobel Prize for lithium ion batteries. If you just point to the Wikipedia page, you'll get thousands of characters. The actual excerpt that matters for citation and grounding is about 500. So there's about 16 x more tokens when you have to process data directly from Google than when you can extract that data, and those extractions have to be semantically rich. You have to pick a segment of the text that answers the question, and so the query matters. And so what does this actually get us? When we measure with ATLAS, again, to ground ourselves, we've seen that we've improved from about 9% on our closed book test to about 56% when we do a semantic extraction over the Internet and extended knowledge. So not only do we get about five x higher of a score, excuse me, but we also reduce the cost by about 28% due to the token efficiency of the extraction. And so we start to demonstrate quantitatively that improving access to knowledge actually improves performance of the underlying agent. And so, yeah, in general, I think the main takeaway that we've hopefully expressed is that while reasoning capabilities on our frontier models are relatively fixed, a lot of improvement can be done in agentic performance by extending the harness. And giving it access to semantic meaning across the Internet. That's it. Thank you. ## Slides ### 00:00:07 # AI Engineer Code Summit PRESENTED BY Google DeepMind ## // AVAILABLE # LIVE TRANSLATION Scan the QR code to access real-time translation. [https://ai.codesummit.io/translation](https://ai.codesummit.io/translation) [QR code for live translation] ### 00:00:39 # AI Engineer Code Summit ## PRESENTED BY Google DeepMind ### // AVAILABLE # LIVE TRANSLATION Scan the QR code to access real-time translation. [QR code for real-time translation] ### 00:01:04 # AI Engineer Europe 2027 ## ExCeL London ### February 17-19, 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. [ai.engineer/europe](ai.engineer/europe) [Black and white photos of London landmarks including Big Ben, the London Eye, and a grand building facade] ### 00:01:37 # What yo doesn't k ff Pinner · CTO, Exa 37° 37'08''N 122° 22'3 ### 00:02:03 # Search has a new customer Agent searches on Exa, per month ## What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO Exa [https://exa.ai/](https://exa.ai/) [Bar chart showing agent searches on Exa per month: 12.6M in Oct 2024, 315M in Oct 2025, and 1.33B in Oct 2026] ### 00:02:37 # Search has a new customer ## Agent searches on Exa, per month - Oct 2024: 12.6M - Oct 2025: 315M - Oct 2026: 1.33B ### What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO Exa [Bar chart illustrating a significant increase in agent searches on Exa per month, from 12.6 million in Oct 2024 to 1.33 billion in Oct 2026.] [QR code labeled "LIVE TRANSLATION".] ### 00:03:09 # Training data will never be enough - **Training cutoff** - **2-5 months** from cutoff to launch - **Model launches** - **2-3 months** until the next model - **Next frontier model launches** **4-8 months** out of date by the time it's replaced ### What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO exa [https://exa.ai/blog/what-your-ai-doesnt-know-the-latest-trends-in-agent-searches](https://exa.ai/blog ### 00:03:36 # Training data will never be enough - Training cutoff - **2-5 months** from cutoff to launch - Model launches - **2-3 months** until the next model - Next frontier model launches **4-8 months** out of date by the time it's replaced ## What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO **exa** [https://exa.ai/blog/what-your-ai-doesnt-know](https://exa.ai/blog/what-your-ai-doesnt-know) [Timeline diagram illustrating the duration from training data cutoff to model launch and subsequent model launches, totaling 4-8 months of data being out of date.] ### 00:04:09 # Training data is a sliver of the web - 3,100T - Whole web - **510T** - Indexed web - **50T** - Frontier training run ## What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO exa [https://exa.ai/blog/what-your-ai-does-not-know](https://exa.ai/blog/what-your-ai-does-not-know) [Concentric circle diagram illustrating the whole web (grey), indexed web (blue), and frontier training run (red) as progressively smaller portions of data.] ### 00:04:40 # Valuable work runs on outside knowledge ## Coding - APIs - Library names - Vulnerabilities - Changelogs ## Legal - Precedents - Legal Filings - Laws & Regulations ## Due diligence - Public statements - Filings - Product reviews - Market data ### What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO exa [https://exa.ai/blog/what-your-ai-doesnt-know](https://exa.ai/blog/what-your-ai-does-not-know) [Icon representing code above "Coding"] [Icon representing a book above "Legal"] [Icon representing a bar chart above "Due diligence"] [QR code linking to exa.ai blog post] ### 00:05:04 # How ATLAS tests agents on knowledge 1. Find the entities 2. Enrich every column 3. Score each row ## THE TASK Which drivers had a car failure at the 2026 Azerbaijan Grand Prix? ### What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO exa [https://exa.ai/](https://exa.ai/) [Table with columns for driver, birthplace municipality, sporting country's government head, and first-points circuit] ### 00:05:37 # How ATLAS tests agents on knowledge ### THE TASK Which drivers had a car failure at the 2026 Azerbaijan Grand Prix? ## What Your AI Doesn't Know: The latest trends ### 00:06:05 # Model memory falls off with recency - About half of entities from before 2025 - **<10%** of entities from 2026 ### What Your AI Doesn't Know: The latest trends ### 00:06:34 # Memory alone isn't enough ATLAS score from the model's memory alone, with no search 9% Closed book [Bar chart showing 9% for "Closed book"] ### 00:07:05 # Agents don't search like people ## People on Google | Mean **3** words: `stripe rate limits` ## Agents on Exa | Mean **12** words ### 00:07:37 ### PROBLEM 1 # Keyword search doesn't understand meaning `companies that **build machines** to **take apart** used **electric car batteries**, founded after 2018, ### 00:08:05 ### PROBLEM 1 # Keyword search doesn't understand meaning > companies that build machines to take apart used electric car batteries, founded after 2018, outside the US - What ### 00:08:36 ### PROBLEM 3 ## Relevant facts are buried in long pages `(6k + 16k + 16k + 16k + 16k + 16k) × 10 = 860k` results page | 5 web pages, as raw HTML | searches | tokens per task The agent reads all of it to pull out a handful of facts. What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO exa [https://exa.ai/blog/what-your-ai-doesnt-know](https://exa.ai/blog/what-your-ai-doesnt-know) ### 00:09:10 # Keyword search isn't enough Adding keyword search is an improvement but overall score is still weak. [Bar chart comparing "Closed book" at 9% and "Keyword search" at 28%] ### 00:09:38 # Understand meaning, not keywords ## PROBLEM 1 · KEYWORD SEARCH DOESN'T UNDERSTAND MEANING `companies that **build machines** to **take apart** used electric car batteries ### 00:10:09 # Every page and query in one embedding space ### PROBLEM 1 • KEYWORD SEARCH DOESN'T UNDERSTAND MEANING ### BUILD THE INDEX • ALWAYS ON - Crawl ### 00:10:37 ### PROBLEM 2 • HIGH-VALUE KNOWLEDGE IS WALLED OFF # Serve more than just the public web ### PUBLIC **Public web** Crawled and embedded by Ex ### 00:11:09 ### PROBLEM 3 - RELEVANT FACTS ARE BURIED IN LONG PAGES ## Serve sourced answers instead of pages ### Highlight > In 2019, John B. Good ### 00:11:38 ## PROBLEM 3 - RELEVANT FACTS ARE BURIED IN LONG PAGES # Serve sourced answers instead of pages ### Lithium-ion battery > In 2019, John B ### 00:12:09 # Same model. Better knowledge. +56% improved ATLAS score over keyword search 28% lower cost ### What Your AI Doesn't Know: The latest trends in agent searches ### 00:12:36 # Same model. Better knowledge. - **+56%** improved ATLAS score over keyword search - **28%** lower cost ## What Your AI Doesn't Know: The latest trends in agent searches Jeffrey Pinner / CTO exa [Bar chart showing 9% for Closed book, 28% for Keyword search, and 45% for Semantic search] [QR code for live translation] ### 00:13:03 # AI Engineer Code Summit ## What Your AI Doesn't Know: The latest trends in agent searches Presented by Google DeepMind Jeffrey Pinner / CTO exa [https://www.exa.ai/](https://www.exa.ai/) ### 00:13:41 # AI Engineer Europe 2027 ## ExCeL London February 17-19, 2027 Europe's flagship technical AI conference returns to ExCe ### 00:14:05 [Abstract background of dark grey dots arranged in a wavy, undulating pattern] ### 00:14:37 # Code Summit ## AI Engineer ### 00:15:02 # AI Engineer Code Summit - LIVE FROM SF//CA [Stylized dark map of a city grid in the background] ### 00:16:06 # WELCOK< HB 37° 37' 08" N 122° 22' 30" W [Stylized dark map with grid lines in the background] ### 00:16:35 # AI Engineer Code Summit ## PRESENTED BY Google DeepMind // AVAILABLE # LIVE TRANSLATION Scan the QR code to access real-time translation. [https://events.google.com/codesummit/translation](https://events.google.com/codesummit/translation) [QR code for real-time translation access] ### 00:17:31 # Code Summit ## AI Engineer ### 00:18:06 ### AI Engineer Code Summit Presented by Google DeepMind ## // AVAILABLE # LIVE TRANSLATION Scan the QR code to access real-time translation. [QR code linking to https ### 00:18:32 # AI Engineer Code Summit ## STAY IN THE LOOP SCAN THE QR CODE OR VIEW THE SCHEDULE AT AI.ENGINEER/SCHEDULE [https://ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to ai.engineer/schedule] ### 00:19:06 [Dark background with a wavy pattern of small grey dots] ### 00:19:37 # AI Engineer Europe 2027 ## ExCeL London ### February 17-19, 2027 Europe's flagship technical AI conference returns to ExCeL London for three days, February 17-19, 2027 — bringing the world's leading AI labs, platforms, and tools to Europe's builders. [ai.engineer/europe](ai.engineer/europe) [Black and white photo of a grand building in London] [Black and white photo of the London Eye and a river] [Black and white photo of Big Ben and the Houses of Parliament] ### 00:19:59 # AI Engineer Code Summit ## STAY IN THE LOOP SCAN THE QR CODE OR VIEW THE SCHEDULE AT AI.ENGINEER/SCHEDULE [https://ai.engineer/schedule](https://ai.engineer/schedule) [QR code linking to ai.engineer/schedule]