# Autoresearch for Dense Retrieval: Test-Time Compute with Frozen Embedding Models — Han Xiao — session 2026-07-01T18:10:00.000Z → 2026-07-01T18:30:00.000Z _63 transcript lines · 41 slides · source: full recording_ ## Transcript The surprising insight is that the most evaluation data often comes from real users interacting with real systems. Now let's talk about the offline values. So offline evaluation still matters. But the methodology changes. Instead of evaluating prompts, we evaluate scenarios. For example, a customer support workflow, a cogention workflow, a research workflow. The agent operates inside the simulated environment. We measure the task completion rate, tool correctness, planning quality, resource usage, which is which becomes exponentially high at high scale. The key takeaway, agent evaluation should be scenario driven, not prompt driven. Once a system reaches production, every interaction becomes a signal. This is Not the biggest shifts in evaluation thinking. Oh, alright. Alright. So can everyone see the, slides? Oh, nice. Alright. So good morning, everyone. Thanks so much for being here. My name is Hanshia. I founded around Jina AI since, 2020 to 2025. And last October, we were acquired by Elastic. So I'm running model inference and the training teams there. And, so here's a question I want to answer today. So big models get syncing better by at inference time. Right? So we call that test time compute. And can a small retrieval model do the same thing? Right? Can it get better by thinking harder at inference, without making the model any bigger? To find out that, I let the agent run auto research overnight. And the answer turned out to be more interesting than a yes or no. Right? So let me show you what I found out. So first Me say what test time compute is. So the idea is very simple. So instead of training a bigger model, you spend more compute at inference time. So you get better answer back. It shows up in very familiar forms, such as best of in sampling, self consistency, or verifiers that re rank the candidates. So Noam Brown from OpenAI, put a number on this. He found that a poker bot, syncing for twenty seconds, got the same boost at scaling the model for 100,000 times. So that's the promise of test time compute. So the real question for us here is, does this promise also for the also hold for search? So here's the reframe that turns this into a retrieval talk. Search is already test time compute. So think about what you do when you build search. You take a trained embeddings, a trained re ranker, some multivector retriever, and a query expander, and then you wire them into a pipeline. So you're spending inference to buy relevance, and you are not reaching for bigger model. You're basically assembling more search at test time. So the real question isn't whether your model is big enough. It is how much pipeline can you assemble at inference and whether that pays off. So there are two versions, two ways to build that pipeline, and I will show you both. The version one, the first one, version a, is the one I will go deep on. So here, an agent writes a little program over a single frozen embedder or encoder. It might chunk the document, to this score. Fuse with different scoring strategy and feeds the results back. So think of think of it as a multi path, algebra over embeddings. The second one, version b, I will come to later. So there, a small agent wires up the retrieval tools, like grab, embed, re rank over a corpus given a fixed, token budget. So it's the same idea implemented at two different levels. So let's start with version a. So version a runs over a small frozen encoder. So there's a common belief is that small models cannot improve there, and test time compute exclusively belongs to the big reasoning models. But let's look at where today's embedders come from. Models such as e five, Mistral, Queen three, embed embedding Gemma, and even our own g nine embedding e five, they all distill from the large language model backbones. So that's the dominant recipe today. And if test and compute leave In the error on representation space, then this detailed model should somehow inherit it, or do they? So that's exactly the question I want to find out. So here's the intuition of, for how a frozen model or frozen better could improve at test time. Let let's look at the three panels. Let's go from the right, left right to the left. So we go from the simplest way to score a match on the left and to the most detailed way on the right. The left, you have a single cosine distance, which is basically one vector per document and one per query. So that's a frozen cosine baseline. On the right, you have this covert style latent interaction where every query token is matched against every document token. So one can consider Cobalt as an extreme case of test time compute. The interesting part is, of course, is that it is in the middle panel where I have online in blue. So you can take a frozen, say, you can take this frozen, say, It's just actually Python program over, the encoder. And the one piece that matters is this embed function. So that's a compute budget. So every function call there basically re embeds some text or switch the LoRa adapter or picks smaller dimensions. So one call is one unit of compute. There are some other constraints, such as the program cannot introduce any hyperparameters, cannot do task routing, cannot add external models, of course. So this constraint those Strings, force the agent to find task agnostic programs instead of a config that's secretly optimized for each task. Then comes the evaluator. So every program runs the same 14 evaluation tasks or discovery tasks, spanning legal, financial, long document, long context, or general retrieval problems. We score it by delta n ECG against the, cosine baseline plus some cost ratio. I will introduce the cost later. Now here's a design choice that matters the most. The loop only ever sees these 14 tasks, and there are 19 more held out tasks. The loop will never touch them or see them. So later, we can ask a very clean question that does what we win here, also hold up there. So and the whole gap the gap is basically the whole experiment. The last part is memory. So it is a simple JSONL, file with one row per program. Each row stores the scores, the cost. The parents, and a short lesson. So the proposal reads this file before every round, and the whole search compounds compounds over time. But compounding has both ways. Right? It builds a real win, of course, but it also also compounds whatever bias, from the objective. And the bias metric does not only mislead one program, it steers the entire family. So now let me set up the models that we use here. We run the search on the single encode. Which is the Jena vFAM nano, only 200,000,000 parameters, state of the art on multilingual retrieval. And we choose nano mostly because the discovery phase as a discovery phase model, mostly because it is small and therefore reduce the cycle time of each experiment. We hold out the bigger model, from the same family plus the unseen families, such as the Gemma model and the Queen model. They share no training data, no backbone, no tokenizer with the Discovery model. We also Out the 19 evaluation tasks as I talked before. And this one those 19 tasks, the loop never sees. So when programs get discovered in this loop, it has to generalize over all encoders and all all 19 tasks. So now before showing any results, let me define the cost of the test time compute. It comes down to one just just one number, c, which is the number of extra forward passes through the encoder. So let me explain With two cars on the slides. They do the same thing, but they they kind of mix in some neighborhood information and then rescore it. The card on the left is what I call the soft centroid. It averages the document, vectors that you already computed, and so there's no extra forward passes. That means its cost c is just one. The card on the right is the first sentence. It re embeds the first sentence of the top top document, which is a brand new forward path. So there, c is greater than one. So while we use the geometry that we already have, the other spans compute on the new path, on the new text. So now that we can prise the compute, we run that exact same loop onto two different rubrics. The first is compute rubric. It animates a program only if the in domain performance beats every program before it. So it is actively pushed to spend more compute at inference time. Second is the transfer Rubik. So it keeps the program only if it improves over over the validation set with nothing getting worse, and it gets no reward at all for spending compute. And to be clear, the validation set is, still comes from what the loop can see. So neither Rubik ever touched the 19 final evaluation task or final holdout task, and unseen encoder. So that's a two brick a two Rubik running under the same loop. So let's see what each one come up with. So Let's first look at the compute rubric. So when you tell it to spend more compute, it draws this very beautiful clean curve. So the x axis is the compute you spent on the log scale and y axis is the score. There are in total 144 programs, and 12 of them sit on the parent front. The cost running from just one, all the way up to almost 15 times. And the in domain score climbed nicely. It it more than triples across that front. So this looks exactly like Test time computes scaling, more compute, more quality. So if I stop here, you will be solved. But this is still in domain performance. We haven't run this experiment on held out, dataset. So let's take a quick look on these 12 programs and run them on the held out, data. So here are the 12 programs drawn as little diagrams. So don't have to, you don't have to read into each one. The only thing that I want you to take away from this is That they are all training free recombinations of the same frozen embedding models. Right? Just chunking, scoring, feedback, and fusion. The cost climbed nicely, steadily from left to right and does look like a clean, scaling story. But the improvement on the held out dataset, as you will see, is not. So now, we run those 12 programs on the held out dataset. And same chart as before, compute runs From the left to right, and scores runs up and down. So the dashed line across the middle is a baseline, and look at the pink line. It, the compute rubric is basically flat, hugging zero all the way out. So out of domain, more compute buys you essentially nothing. Now look at the blue dots, which is the transfer programs. They all sit on the left because they are cheap, and everyone is above the pink line. So the cheapest one only has, like, has zero extra compute. It still be the most Expensive program. So more compute did not transfer. The cheap structure did. So if we plot every program against every held out, task, we get this heat map. The four blocks are the four encoders, and three of them we have never seen in the discovery phase. In each block, the rows are the programs and the column are 19 evaluation tasks. Green means an improvement. Red or pink means a drop. The picture is generally mixed. Helps about half of the sales, but improvements are uneven. So on I on average, it comes out flat. Compute does help in places, but it doesn't help reliably across all new all new tasks and all new encoders. So now let's look at these, look at the other rubric, the transfer rubric. It picks the six completely different programs, and they are all very cheap and most one and a half times, more compute than the cosine baseline. The best one wins 83%. On the held out dataset, and it never loses on a single task. So now what what do this program, actually do? So they only test some query and document vectors that you already have, and they add a little cheap math on top of that. Some nudge the query towards a document it already likes. Some pick a few directions and, in the space and rescore, along those directions. So they are very small structure change, but enough to pull the document, the right document up. So it's all a combination, no new models. And this really transferred to across models and languages. Remember, in the discovery phase, we only use JINA embedding, a JINA v five nano, and but the improvement is positive across all four encoders. And the biggest bar is on the JAMA and the QUIN. So those on the, two families it never sees. So this is isn't some quirk of one model is general is right on general embedding geometry. So that was version a. A frozen encoder with very cheap structure, and it scales, but low compute, doesn't scale. And auto research is how we found that. But let me move one level up, from the model layer to the search pipeline. And you will see the same test time compute reflect in the pipeline level. In 2025, we have this deep research, and agentic search product, which was basically just a one loop over the, open web. Twenty twenty six, we move to a long horizon task, which adds implementation, sandbox, evals on top of the retrieval and running for hours. So both patterns need more looping and more compute at test time. So studying this genetic search at test time, I built three open source projects for that. The first one is data room. So you give a token budget, it searches, it reads, it writes. So, over and over until it packages everything into a zip file. So, I coded Room because it somehow reminds me, like, prepare the data room for the investors, back when I was a founder. So that zip file distills a corpus on the, is you can you can imagine this, zip file is a distilled corpus of the open web ready for the next agent or large language model to consume. And notice the token economy here. So you are basically exploring the web and building the corpus using very cheap tokens from small language models. And then you save the expensive frontier tokens for later for Quotation. The second one is search box. So this is a test bed to study agentic search and two calling. It is designed it is designed to be air gapped so the agent have no Internet access. It's basically like you lock the agent in a room or in a box, and you give it a data room and ask questions about it. So to answer those questions, the agent has to assemble a search pipeline at test time. A pipeline made of local tools, things like a grep, embed. Rerank. And this allows you to explore some very interesting research questions such as, which tool does the agent reach for first, or is Grab all you need, or does forcing more compute help on hard questions, Or will the agent build up a search pipeline that it will reuse later? So SearchBlocks is a testbed to explore those research questions. So but how do you evaluate, a generic search like that? Well, you need hard questions. That's basically the third project is knowledge graph. So it turns a corpus or data room into a knowledge graph, and every fact become an edge and linking from subject to an object. Then we can work on the longest path through that graph, and those long chains become multi hop questions then that no single passage can answer. So the agent has to spend more test time compute connecting the fast to get there. So it's also the tool for Doing a private verifier. So let's connect all the dots together. So I introduced two versions of test time compute for search. Both versions are doing the same thing. They are spending more compute and test time, and neither of them grows the model. ## Slides ### 00:00:16 # The Evaluation Signal Hierarchy - **Benchmarks** High volume, low operational value. The foundation, not the destination. - **Scenario Evals** Targeted workflows. - **Production Telemetry** Low volume, maximum signal value. [Pyramid diagram showing a hierarchy of evaluation signals. The base is "Benchmarks", the middle is "Scenario Evals", and the top is "Production Telemetry". An arrow pointing down on the left is labeled "Volume", and an arrow pointing up on the right is labeled "Operational Value".] ### 00:00:48 # Offline Evals: Scenario-Driven Simulation ## Agent Sandbox Flow - Test Runner - Simulated Tools - Execution Steps - State Update - Discrete Outputs ### Metrics - Completion Rate: 98.5% - Tool Correctness: 100% - Plan Quality: High - Simulated Cost: $0.05 Scenario-driven, not prompt-driven. [Flowchart showing a Test Runner feeding into an Agent Sandbox, which contains Simulated Tools, Execution Steps, and State Update, with feedback from State Update to Execution Steps. The Agent Sandbox outputs to Discrete Outputs, which then leads to a Metrics table.] ### 00:01:17 # Engineering the future of AI Presented by Microsoft - yugabyteDB - Daytona - Temporal - baseten - Amazon AGI Lab - Microsoft - Light ### 00:01:50 ## Autoresearch & test-time compute for information retrieval Han Xiao VP of AI, Elastic @hixiao • in/hixiao87 EMBEDDING GEOMETRY ### 00:02:20 ## TEST-TIME COMPUTE ### Spend more compute at inference, get a better answer. Trade inference compute for quality: rather than a bigger model, the model does more work per query, along a smooth cost-versus-quality curve. - **BEST-OF-N**: sample many candidates, keep the best. - **SELF-CONSISTENCY**: majority-vote over sampled chains of thought. - **VERIFIER SEARCH**: spend inference FLOPs, climb a Pareto curve. > Having a bot think for just **20 seconds** in a hand of poker got the same performance boost as scaling up the model by **100,000x** and training it for 100,000 times longer. > Noam Brown, OpenAI, 2024 Autoresearch & test-time compute for information retrieval ## Engineering the future of AI ### 00:02:49 ### THE REFRAME ## Search is test-time compute. You wire trained embeddings, rerankers, multi-vector retrievers, and query expanders into a **pipeline at inference** to squeeze out relevance. You do not need a bigger model. You assemble more search at test time. **DON'T SCALE IT** one keyword match. Cheap, and probably not good enough. **SCALE IT** embed, retrieve, rerank, expand, fuse. More inference can buy more relevance. The question is not whether the model is big enough. It is how much search pipeline you assemble at test time, and whether it pays off. Autoresearch & test-time compute for information retrieval ### 00:03:19 ## Two ways to manufacture that pipeline at test time. ### TWO VERSIONS A. RECOMBINE THE GEOMETRY An agentic loop writes **programs** over one frozen encoder: chunk, z-score, fuse channels, feed back. The pipeline is multi-pass embedding algebra. this talk's core B. COMPOSE THE TOOLS A small agent LLM wires **retrieval tools** (grep, embed, rerank, select_diverse) over a corpus under a budget. The pipeline is an agentic tool chain. searchbox · later Autoresearch & test-time compute for information retrieval ### 00:03:46 ### TWO VERSIONS ## Two ways to manufacture that pipeline at test time. ### A RECOMBINE THE GEOMETRY An agentic loop writes **programs** over one frozen encoder: chunk, z-score, fuse channels, feed back. The pipeline is multi-pass embedding algebra. this talk's core ### B COMPOSE THE TOOLS A small agent LLM wires **retrieval tools** (grep, embed, rerank, select_diverse) over a corpus under a budget. The pipeline is an agentic tool chain. searchbox · later Autoresearch & test-time compute for information retrieval ### 00:04:17 # "Small models can't improve." But they are distilled from LLMs. Test-time compute is said to belong to large reasoning models, with small models unable to improve. Yet today's embedders - including the small deployable ones - are distilled or adapted from LLM backbones: - **Mistral-7B** -> E5-Mistral, SFR-Embedding, GritLM, NV-Embed, bge-en-icl - **Qwen3** -> qwen3-embed, jina-embeddings-v5 - **Gemma 3** -> EmbeddingGemma - **LLaMA** -> RepLLaMA, LLM2Vec The lineage from a large language model to a small deployable encoder is now the dominant recipe, not the exception. If test-time compute lives in the LLM representation ### 00:04:51 ### THE INTUITION ## Scoring runs from one cosine to late interaction. ### SINGLE-VECTOR COSINE - `cos(q, d)` - one vector per document - ### 00:05:18 # Engineering the future of AI ## THE INTUITION # Scoring runs from one cosine to late interaction. ### SINGLE-VECTOR COSINE - `cos(q, d)` ### 00:05:49 ### THE INTUITION ## Scoring runs from one cosine to late interaction. ### SINGLE-VECTOR COSINE - `cos(q, d)` - one vector per document - **FROZEN BASELINE** ### SENTENCE MAXSIM - `max_s cos(q, s)` - same encoder, document split into sentence vectors - **TEST-TIME COMPUTE** ### COLBERT MULTI-VECTOR - `sum_i max_j cos(q_i, d_j)` - every query token, max over all doc tokens - **NEEDS A MULTI-VECTOR MODEL** Autoresearch & test-time compute for information retrieval # Engineering the future of AI [Diagram showing a single vector `q` connected to a single vector `d`] [Diagram showing a single vector `q` connected to multiple sentence vectors `s`, with one `s*` highlighted as the maximum] [Diagram showing multiple query vectors `q_i` connected to multiple document vectors `d_j`] ### 00:06:22 # How much can a frozen single-vector encoder improve at inference alone? **NO RETRAINING** **NO AUXILIARY MODEL** **NO LEARNED PARAMETERS** **ONE ### 00:06:49 ## AUTORESEARCH # Autoresearch: let the agent climb the hill. An agent runs the research loop itself: change one file, run a short fixed-budget experiment, keep the change if the metric improved, otherwise revert. Repeat, overnight. It is hill- ### 00:07:21 ## Autoresearch: let the agent climb the hill. An agent runs the research loop itself: change one file, run a short fixed-budget experiment, keep the change if the metric improved ### 00:07:45 ## METHOD An LLM agent writes programs over the frozen encoder, and an evaluator scores them. - **Proposer**: LLM agent, Opus 4.6 - ** ### 00:08:19 # The proposer is an LLM, used as the mutation function. ## METHOD 1: PROPOSER - **DESIGN** Opus 4.6 reads the current frontier source and the memory file, edits **one** Python program, and proposes the next. Keep it if the metric improves, else revert. No human in the inner loop. - **CAVEAT** It optimizes **exactly** the metric you hand it - not the one you mean. Reward in-domain performance and reward spending compute, and that is what it will chase. Whether the improvement survives out of domain is a separate question. The objective is the steering wheel. Autoresearch & test-time compute for ### 00:08:53 ## The program is arbitrary Python over the frozen encoder. ### METHOD • 2 PROGRAM ### DESIGN ```python rerank(q_emb, d_emb, sim, *, embed_fn, q_texts, d_texts) -> scores ``` `embed_fn` is the **test-time-compute budget**. It re-embeds any text, switches the LoRA adapter (retrieval / passage / classification / matching), picks a Matryoshka dimension, or caps length. One call is one unit of compute. ### CAVEAT Auto-rejected: hyperparameters, stochastic ops, task-specific routing, external models, learned parameters, trivial constant-only variants. These constraints force **task-agnostic structure**, not a per-task tuned config. Autoresearch & test-time compute for information retrieval [Flow diagram showing a process: Proposer (LLM agent, Opus 4.6) feeds into Program P (Python over j-v5-nano), which feeds into Evaluator (ΔnDCG, 14 tasks), then Memory (JSONL file), and finally Registry (144 programs).] ### 00:09:18 # METHOD · 3 EVALUATOR The evaluator scores every program on the same 14 tasks. ## DESIGN Every program runs on the same **14 MMTB ### 00:09:50 ## METHOD · 3 EVALUATOR # The evaluator scores every program on the same 14 tasks. ### DESIGN Every program runs on the same **14 MMT ### 00:10:18 ### METHOD - 4 MEMORY ## Every program is logged, and the log conditions the next one. - **Proposer**: LLM agent, Opus 4.6 - **Program P**: Python over j-v5-nano - **Evaluator**: ΔnDCG, 14 tasks - **Memory**: JSONL file - **Registry**: 144 programs ### DESIGN A JSONL file, one row per program: per-task ΔnDCG@10, win/tie/loss, cost ratio, parent, a novelty claim, and a post-hoc lesson. The proposer reads the frontier source plus this history, so each round builds on the last. ### CAVEAT Compounding cuts both ways: memory builds on real wins, but it also compounds whatever bias the objective has. A biased metric does not just mislead one program, it steers the whole lineage. Autoresearch & test-time compute for information retrieval [Flow diagram showing a sequence of steps: Proposer -> Program P -> Evaluator -> Memory -> Registry. A dashed line indicates feedback from Memory back to Proposer.] ### 00:10:50 ### SETUP ## What we search on, and what we test on. - **DISCOVERY - THE LOOP SEARCHES HERE** - `j-v5-nano` - ### 00:11:18 ## What we search on, and what we test on. ### SETUP **DISCOVERY - THE LOOP SEARCHES HERE** - **j-v5-nano** - ### 00:11:51 ## THE DISTINCTION # Cost is one number: extra forward passes through the encoder. `c = 1 + (extra forward passes / baseline passes)` ### TEST-TIME COMP ### 00:12:22 # Engineering the future of AI ## Two Rubrics, Same Loop ### We run the search under two rubrics. **COMPUTE RUBRIC** - Admit a program if its ### 00:12:43 ## TWO RUBRICS, SAME LOOP ### We run the search under two rubrics. **COMPUTE RUBRIC** Admit a program if its in-domain performance beats every prior one. The proposer is encouraged to spend more inference. → how high can the in-domain frontier climb? **TRANSFER RUBRIC** Admit only if a validation split improves with no task regressing past 0.05. No reward for spending compute. → what holds up out of domain? Neither search ever sees the 19 held-out evaluation tasks or the unseen encoder families. Autoresearch & test-time compute for information retrieval ### 00:13:19 ### WHAT COMPUTE BUYS, IN-DOMAIN ## Told to spend compute, the search draws a clean Pareto curve. - **144** programs searched over 144 generations - **12** Pareto-optimal, from c = 1.2 to 14.7 - In-domain mean ΔnDCG@10 climbs from +0.07 to +0.24. This looks exactly like test-time-compute scaling. - **In-domain only.** The held-out test comes next. Autoresearch & test-time compute for information retrieval [Scatter plot showing "in-domain mean ΔnDCG@10" on the y-axis versus "cost ratio c (x baseline encoder passes, log)" on the x-axis, with a pink line connecting 12 Pareto-optimal points and individual black circles representing 144 searched programs.] ### 00:13:50 # WHAT COMPUTE BUYS, IN-DOMAIN ## Told to spend compute, the search draws a clean Pareto curve. - **144** programs searched over 144 generations - **12** Pareto-optimal, from c = 1.2 to 14.7 - In-domain mean ΔnDCG@10 climbs from +0.07 to +0.24. This looks exactly like test-time-compute scaling. - **In-domain only.** The held-out test comes next. Autoresearch & test-time compute for information retrieval [Scatter plot showing in-domain mean ΔnDCG@10 versus cost ratio c, with a pink line highlighting 12 Pareto-optimal points among 144 searched programs.] ### 00:14:19 ### WHAT THE SEARCH WROTE # Twelve programs on the compute frontier. - Bidirectional ZScore c=1.2 - SentMaxSim c=2.2 - AdaptGran ### 00:14:45 # AI Engineer World's Fair ## KEYNOTES - LIVE STREAMING JULY 1 - + AUTORESEARCH [Illustration of a dark, cracked robot with golden lines, holding a scroll and a quill] ### 00:14:49 ### WHAT IT BUYS ON HELD-OUT DATA ## On unseen encoders, compute is flat. Cheap structure is not. - **14.7x** the most compute ### 00:15:19 # THE COMPUTE FRONTIER, CELL BY CELL ## Compute helps about half the cells, and collapses the rest. Each row is one of twelve programs ordered by compute cost ### 00:15:49 ## THE TRANSFER RUBRIC # A different rubric finds six cheap programs that transfer. | PROGRAM | c | MEDIAN | WIN-RATE | MEAN | WORST CELL | | :------------------ | :--- | :------- | :------- | :------- | :--------- | | ZeroCost-Consensus | 1.00 | +0.0051 | 68.4% | +0.0148 | -0.1070 | | Deca-Axis | 1.46 | +0.0051 | 68.4% | +0.0133 | -0.0365 | | Dodeca-v077 | 1.46 | +0.0046 | 66.7% | +0.0126 | -0.0420 | | **Transpose-Consensus** | **1.50** | **+0.0043** | **83.3%** | **+0.0219** | **0.0000** | | Cross-Axis | 1.46 | +0.0025 | 64.9% | +0.0097 | -0.0240 | | Penta-Axis | 1.46 | +0.0019 | 59.6% | +0.0107 | -0.0485 | Autoresearch & test-time compute for information retrieval [Table comparing six programs with metrics like median, win-rate, mean, and worst cell, each program having a small diagram next to its name.] ### 00:16:23 ## THE TRANSFER RUBRIC # A different rubric finds six cheap programs that transfer. | PROGRAM | c | MEDIAN | WIN-RATE | MEAN | WORST CELL ### 00:16:51 # Improvements are largest on encoder families never seen in discovery. ## IT TRANSFERS Discovered on j-v5-nano. Mean ΔnDCG@10 is positive on all ### 00:17:22 ### THE TREND # Test-time compute is the pattern for deep research and long-horizon tasks. ## 2025 · Deep research ## 2026 · Long-horizon tasks Research and execution want different tokens: build the **dataroom** with cheap local ones, and save the costly frontier budget for the execution that actually needs it. That two-tier split is dataroom, then **searchbox**, next. Autoresearch & test-time compute for information retrieval [Flowchart illustrating a "Deep research" process: Query leads to a loop of Search, Read, Reason, which then leads to Answer. This loop is described as "loop until budget", "one web-bound loop", and "all Web IO".] [Flowchart illustrating a "Long-horizon tasks" process: Query leads to Search, Read, Reason (labeled "research"), then to a **dataroom**, which leads to Read, Run, Write (labeled "agent loop"), and finally to Output. This process is described as "phase 1 research Web IO", "build the corpus", and "phase 2 runs offline NO Web IO".] ### 00:17:45 ## THE TREND # Test-time compute is the pattern for deep research and long-horizon tasks. ### 2025 · Deep research loop until budget Query -> Search ### 00:18:17 ### VERSION B - THE CORPUS STAGE # dataroom: a loop for knowledge dump. Given a token budget, spend it on **local small models** instead of a frontier model ### 00:18:48 ### VERSION B - THE SEARCH STAGE # searchbox: a testbed for studying agentic search loops. An airgapped testbed for search as test-time compute: lock ### 00:19:20 VERSION B • THE SEARCH STAGE # searchbox: a testbed for studying agentic search loops. An airgapped testbed for search as test-time compute: lock an agent in the box with one `.zip` dataroom and no web, so it can only answer by composing its own pipeline from local tools - grep, embed, rerank, similarity, cluster, select_diverse. Nothing leaks in; the search has to exhaust what is ### 00:19:50 # knowledge-graph: hard multi-hop questions for a private verifier. — VERSION B • THE VERIFIER Trivial questions are useless for agentic search: if one grep finds the