# Memory Harnesses for Long-Running Research Agents — Stefania Druga — session 2026-07-01T18:40:00.000Z → 2026-07-01T19:00:00.000Z _57 transcript lines · 40 slides · source: full recording_ ## Transcript They are the finish line. We spent three years making The protocol is smaller and simpler. Every time we ran into a problem, we ask what can we take away? What abstraction can we erase? What property can we remove? What relationship can we break? The result is a very small protocol centered around two objects, a durable promise and a durable task. That simplicity matters because even simple concurrent distributed protocol have a complex state and behavior space. So in other terms, implementing even simple protocols on top of a few simple primitives is tough. Let's make this concrete with NATS. NATS gives us a Hello. Welcome. This is a big room, so you're if you're in the back, don't I'm hesitate to come closer. My name is Stefania Drouga. I'm a research scientist at Sakana AI in Tokyo. I used to be based here, and AI engineering, is home community for me before being the Hyperloop, So it's very good to be back. And today, I'm going to talk to you about memory harnesses for long running research agents on device. So if you work with long horizon tasks, you probably ran into This issue of context bloat. Right? Like, when the model starts contradicting itself or it has to redo the work because it forgot it did that task in the first place, or it starts to drift from your questions because it forgot them. And this this matters now more than ever because from these recent projections from METER, we see that the trend is to solve longer and longer, horizon tasks and also that we're Getting fewer and fewer model releases. So at some point later this year, we're gonna have this convergence, right, where we'll get many more long term horizon tasks and fewer model releases. So that makes this issue of dealing with context rot a priority. And why did I wanted to to tackle this problem on local models and with a local harness? Maybe some of you have seen this tweet. It's only two days old. The CEO of Actually shared how their company managed to reduce their AI spent while actually increasing, the AI usage. And the way they did that was by transitioning to use many more local models, but also having better practices, like using better routing, better caching, keeping the context clean, and then having better visibility for what people are using it for what, what kind of task. So We are seeing the local models, like, crossing the line. Right? Like, GLM is on everyone's minds, like, especially with Fable going away. DeepSeq v four Flash can now be run on, m three ultra. And there's still a bottleneck for RAM. It's tricky. But these local models are starting to be useful for agentic tasks and for tool use. So I wanted to show you what has been my setup for the experiments I'm gonna share. Due today. This this is my Mac. It's still running evaluations right now, back in my desk in Tokyo, and I'm controlling it from my phone. And after running Evals nonstop for a couple of days, it started to get hot. So I had my husband put fans around it. We're running out of fans. But the the machine is still running and the evals are still giving results. On this m three ultra with 96 gigabytes and 28 core CPUs. I'm using two models. I'm using the Gwen27b quantized at four bit and the deep seek b4 flash. And before I show you how I build the memory harness on this machine, I wanted to tell you what this what is this an example of? Right? Like, memory, when we design a harness for memory, this is the mental model I want you to have in mind. You can think of memory as a write manage read. Loop. So it's not just the database store. It's actually this control loop around the model. More concretely, how did I take that loop and customize it? So this is my harness design. Like, I started with research agents that are the small agents because they have zero durable memory, and I wanted all the memory to come from the harness. And then, in the middle, I have a core, which is always shown to to the agent, of traces. And then I have a recall. Block where I'm testing different modes and an archival block where I'm keeping track of information across different, sessions. And in that recall block, I'm actually going through a ladder of modes that I'm testing. The baseline is, like, not to use memory at all. No recall at all. So I'm testing for that. Next is to use rag vector vector rag, just to see whatever, like, Harness with pull in terms of similarity. Then it's to use a decisions, ledger where I actually keep track of what decisions are being made for every turn and then I can prioritize them. And last but not least, and this piece is very important, I have what I call an oracle, but basically this is the ground truth. So this is like telling the harness for every loop what the correct memory that needs to be retrieved is. And the model is fixed across all the different tasks. So the only things that I'm changing is these different variables in the recall block. And I wanted to give you an example of a first task that I tested. So I wanted to see if I give the agent a task of doing literature review, and I'm including a lot of papers in the corpus where there was a big scientific claim. Like, this is actually a nature paper where they said they Discovered 742,000 promising materials. Like, it was a very big claim which got retracted later. But the retraction, it's a much smaller, like, haystack needle in that corpus than the headlines and the citations. So I wanted to see if if the system can retrieve the right answer for these type of questions. And what I found was because, like, for these stuff Tasks. All the papers and all the information fit into the context. The memory actually didn't add more capability. It was the same performance with memory and without memory, and it only added more cost. So when your task fits in context, the harness doesn't add much. However, if I start to run tasks that are longer term horizon and the entire task and the Relevant context doesn't, fit, then having a good memory harness really starts to pay off. So this is another example of a task that I ran. This is actually from an established benchmark for a long horizon, tasks memory. It's It's called Xbench. And this is an example of a question. Right? So I'm asking a question and then, like, the right answer is, you know, like, step one twenty four. But the moment when I ask the question, I'm asking it, like, at step 500. So it's completely outside of the context window, and the model needs to use the memory harness to retrieve the specific answer from the right step. So I'm testing this by changing the different policy ladder that I explained before with memory off, by deploying different types of recall, and by using the oracle as a reference. And what I found was that with the ranked recall, the model gets the right answer, more frequently than without. And here is a breakdown of the decomposition of performance on this Xbench tasks. So I ran over, 68 questions. And for each of these questions, there were, like, multiple, cells and lots of different seeds. And What I found was that the rank only ledger performed the best. And it performed better than, like, just gating the harness by saying, do you need to use memory or do you not need to use memory? And you're probably going to ask, like, why is the oracle not hitting, like, the max? And I'm going to explain that too. So the oracle, what it does, it provides the right information, the right memory to the model, but it doesn't force To use it. So the model can get the right memory, but still retrieve the wrong information, or choose to ignore it, or be confused. So that's why the oracle, in this case, doesn't hit the max performance. And I've done lots of ablations on these tasks to see what happens if I give arbitrary examples, what happens if I give it the wrong step, what happens if I give it the most recent step. And I still found that the best performing Condition was the one with the ranked policy for recall. And this actually works on several models, not only on the QAN27B but also on the DS4 Flash. And it also works across different benchmarks. I also tried it on the Spider v2 benchmark. And it's not just that it gives you better recall. It actually costs less. So maybe a good heuristic to have here is that bad memory is Expensive because it spends more tokens and it can send agent the wrong way. But having a good structural policy for recall can save you a lot of tokens and budget. So one thing that I want to encourage you from this experiment is to consider the recall policy as a first class metric and to start to think about how you might use it in your systems. Like, what are the type of Memories that you want to store, how do you rank them? Like, how do you design your recall function? And then, what are the type what survives when you run this over and over and over And, multiple sessions, multiple runs. And this is just a simple first experiment. But the memory technique landscape is very rich. So there's over 30 runnable cookbooks that are shared. In this open source repository from, Diamond. And memory is complex. We have short term, long term different cognitive techniques. We can use start to use evaluation results as well. And right now, there's actually a pretty broad landscape of solutions. Right? So going from simple file system retrieval to training memory models, there is there is a wide spectrum Solutions from less structural to completely structured. So I think there's a lot of research we're gonna see in this space. It's important. It becomes more and more relevant. And for me, it's been super fun to test this on local models because I got to control everything. I got to control the data I was using, the entire traces of compute and evaluations. And, yeah, I see that in An example of sovereignty. And it comes at a cost. I didn't tell you that these local models, I can only, what, run them in serial. Like, they don't support batch querying for the DeepSeq v4 Flash. So that's why I am still running evaluations back on my computer in Tokyo or I was doing it on the flight on my way here because it takes a long time. But I still think it's very powerful and it's a very good test for what memory can do when you can control Every single step of the pipeline. And this sovereign capability is part of a bigger ecosystem that is very important for us at Sakan AI in Japan. We believe in the importance of sovereign AI today more than ever, And we are also hiring. So if you're interested and want to hear more about this and if you want to come join us in Japan, come talk to me. Thank you very much. Hi, everyone. I am Balash Robert, and today, I will talk to you about what is the last thing that AI will take away from us as people in the software business. So at a point where writing code is no longer the bottleneck, the real thing is figure is figuring out what it is that you should be building. And that comes down to to people skills and being able to work the room because you can't prompt a room. You can prompt your AI. So at the beginning of the year, we held an internal hackathon, where we had about 21 agents, agent Ideas, and 17 of those were abandoned because they actually created no, business value. They, we either didn't have data access or or just didn't make sense to build it. And those four were the ones that actually had a very big impact on how we work today. And it's it's a very good example of of just making sure that we are building what is Earth building. And throughout my career in the past thirteen years, I've always been, the bridge between business and IT and the developers. I started writing, well, initially testings, functional designs, specifications, and then, and then I wrote them. And as, as a functional consultant, I worked with large ERP and CRM programs in The US and The UK, and then I founded Visual And, essentially, I train my my team on how to elicit those requirements in a way, that we can turn them into good, specifications for developers to build, for consultants to configure, and most recently, for AI to build. And what's not really changed over the years is how we interact with our customers. How we interact with systems, how we interact with AI is very much changing. And that's that's, that's the big thing now. But if you can read the room, if you can elicit the right requirements, then you will be able to build more valuable software. And that essentially, the big shift over the past two, three years was that getting access to code and being able to build is no longer the bottleneck to the software development life cycle. Now the real bottleneck is getting your People, your stakeholders, your decision makers into the room and being able to access them and elicit the requirement and being able to spend the time with them. So that's the right that's the real bottleneck, figuring out what it is that should be built. Because you can prompt your code. You can prompt your AI. You can prompt your whole specification. You can't prompt your room. And what a model can't do is very similar to how Henry Ford's analogy of What he said about asking his users or his customers. If he'd asked them what it is that they needed, they would have said they needed more horses. But in reality, he built a car, and he made a very big success on them. So if you're just using AI, to to make things you know, build things better, the chances are that you are replicating what already exists because AI, by definition, is coded to give You the most common answers. So so for us, the real job is to make sure that AI moves away from that average into what is better for us. So we can just get to, not a faster horse, but actually produce a car that's a magnitude shift better than what we had. So it's really an interesting world where being able to write good code is no longer the the most important skill to have. Actually, the real skill now is Coming the analysis analyst toolkit, which is, you know, things like story mapping, business model canvas, value canvas, and those those good old things that we are so used to using as functional consultants, business analysts, or or in in the world of design thinking. So I'd like to zoom in on story mapping because As the the skill set that I found as the most valuable. So, once you have the story map with the backbones and understand at each step what your customers, your users are doing, That would give them the ability to, to move forward, in their in their processes. So, here's a, support systems, user story map, contacting, triaging, resolving, and then essentially closing the Case. With this, you can understand different stages of the process, and then capture the user stories beneath them. It is intended to stay at a fairly high level so you can get a a big picture, And then in you can decide, what it is that you want to build or release one, like capturing intent, classifying urgency, drafting a grounded answer, and then logging logging it to a system of record. That's essentially your MVP. So the first things that you'd want to build, and those are your first four user stories. And beneath those, you've got the, the second set of user stories, like reading a sentiment, writing to a team, suggesting next action, chatting, checking satisfactions, so on and so forth. Those will be part of your backlog. So what would allow you to, to get really good, agentic results is by honing in ## Slides ### 00:00:17 `promise.create(id)` [An arrow points right from beneath the code `promise.create(id)`] ### 00:00:52 - `promise.create(id)` - `task(execute(id))` - Promise - id - `task(execute(id))` - `promise.sett ### 00:01:16 # World's Fair ## AI Engineer Resonate - Queues - Key Value Store - Delayed Messages ### 00:01:51 # AI Engineer World's Fair - Braintrust - Cloudflare - TOPK - together.ai - builder.io - cohere - Amazon AGI Lab ### 00:02:22 # WHY CARE ## Context bloat — the horizon is growing ### Context bloat fails three ways: - Contradictions — reverses an earlier decision - Redo work — repeats ### 00:02:46 # Context bloat — the horizon is growing ## Context bloat fails three ways: - **Contradictions** — reverses an earlier decision - **Redo work** — ### 00:03:18 # Why local models and harness ## AI Spend at Coinbase (Bars) -vs- Token Usage (Line) Left axis = USD spend (stacked by org). Right axis = total company tokens. ### 00:03:51 ### WHY LOCAL | JAN 2026 # Local models are crossing the line On one 96 GB M3 Lutra the newest GLM / DS4 builds are now agent ### 00:04:19 # My setup for this talk work | LANE | MODEL | | :---------- | :---------------- | | Frozen qwen | qwen3.6-27b-4bit ### 00:04:51 ### THE DESIGN PROBLEM # Memory is a write-manage-read loop A long-running agent fails in three ways — it **contradicts** an earlier decision, **redoes** ### 00:05:19 ### WHAT WE BUILT ## Harness Design - Sub-agents - isolated context - Research agent - smolagents - zero durable memory - ARCHIVAL — cross ### 00:05:52 # Harness Design ## WHAT WE BUILT - **Sub-agents** (isolated context) - **Research agent** (smolagents, zero durable memory) - ** ### 00:06:20 ### WHAT WE BUILT # Harness Design - **Sub-agents** - isolated context - **Research agent** - smolagents - zero durable memory ### 00:06:51 TEST MEMORY MECHANISM - GNoME v2 # Answer from the correction or the stale headline? The agent reads a long evidence trail on AI materials discovery. Early records make a big claim; later records, buried deep, qualify it. Then the final prompt asks for the **current best assessment**. ### THE PACKET Early: "found **742,913 promising materials** - **another lab made 23 discoveries**." Later (buried): many were new only to the model's queue, not the field - some were registry-neighbor duplicates. ### GOOD ANSWER ✓ Useful for screening and candidate generation — but the strongest novelty claims should be qualified. The headline count overstated what was proven; the lab's novelty was partly "new to the platform," not new to science. ### BAD ANSWER ✗ Repeats only the early headline: "GNoME discovered hundreds of thousands of new materials" — ignoring the later correction once it has evicted. That regression is exactly the failure this demo exposes. Literature-review task: revise the conclusion as evidence arrives. GNoME's materials claim, reframed by a follow-up and codified by a **Nature Author Correction** in January. GNoME: Merchant et al., "Scaling deep learning for materials discovery," Nature 624, 80-85 (2023), doi:10.1038/s41586-023-06735-9. ## Engineering the future of AI ### 00:07:25 # TEST MEMORY MECHANISM - GNoME v2 ## Answer from the correction or the stale headline? The agent reads a long evidence trail on AI materials discovery. Early records make a big claim; later records, buried deep, qualify it. Then the final prompt asks for the **current best assessment**. ### THE PACKET Early: "**found 742,913 promising materials - another lab made 23 discoveries.**" Later (buried): many were new only to the model's queue, not the field - some were registry-neighbor duplicates. ### GOOD ANSWER ✓ Useful for screening and candidate generation — but the strongest novelty claims should be qualified. The headline count overstated what was proven; the lab's novelty was partly "new to the platform," not new to science. ### BAD ANSWER ✗ Repeats only the early headline: "GNoME discovered hundreds of thousands of new materials" — ignoring the later correction once it has evicted. That regression is exactly the failure this demo exposes. Literature-review task: revise the conclusion as evidence arrives. GNoME's materials claim, reframed by a follow-up and codified by a **Nature Author Correction** in January. GNoME: Merchant et al., "Scaling deep learning for materials discovery," Nature 624, 80-85 (2023), doi:10.1038/s41586-023-06735-9. ### 00:07:45 # When context fits memory just adds more tokens All three configurations get the answer right — 5/5 seeds each. Accuracy is pinned at 5/5 across all three. The deployed memory policy spends ~2x, the orchestrated variant ~3x. [Bar chart showing relative token cost for three configurations: "No memory off" (1x), "Deployed decisions_json" (~2x), and "Orchestrated sub-agents" (~3x)] ### 00:08:27 ### ONE CELL · MECHANISM Xbench Task # How we test memory on long horizon tasks? A long-horizon QA task. A fact is stated once, early — then buried under ~2 ### 00:08:47 # How we test memory under eviction from context When the needed state has evicted but still exists in the ledger: can the harness retrieve the right part for this step? ## TASK SETUP - **COMPONENT** - Long-horizon task: decisions made early, needed later - Eviction: the needed state no longer fits in context - Ledger: prior decisions/evidence still exist in memory - Recall policy: the harness decides which memories to inject - Outcome: did it recover ### 00:09:18 # The XBench decomposition XBench accurate retrieval - n=240 - qwen3.6-27b-4bit - off: 50 - deployed: 70 - gate_only: 70 - rank ### 00:09:52 # The XBench decomposition ## XBench accurate retrieval - n=240 - qwen3.6-27b-4bit - off: 50 - deployed: ### 00:10:20 # Ablations on injected memories ### XBENCH CONTROLS - ISOLATED QWEN REPRO ### What the ablations rule out - Arbitrary, recent & wrong-step retrieval all land at or below the 50/240 floor. - Only sub-goal-repaired recall clears it: relevance 120, need-gated 130. - Oracle ceiling 160/240 (exact-step injection). **LEGEND** - no-memory floor - random controls (arbitrary / recent / wrong-step) - decisions_json (state dump) - Ranked recall — the lift - oracle ceiling XBench accurate retrieval n=240 qwen3-6-27b-4bit (isolated) source: xmodel report [Bar chart showing "passing cells (240)" on the Y-axis against different ablation conditions on the X-axis. Values are: off 50, random 41, recency 35, wrong-step 40, decisions 75, relevance 120 (red), oracle 160 (red outline). A dashed line indicates the no-memory floor at 50.] ### 00:10:52 # Better recall cost less Under eviction, the repair didn't buy accuracy by spending more context — it cut waste by selecting **better** context. - **+45 ### 00:11:19 ## THE AGENDA # From storing memory to learning its authority | MEASUREMENTS RULE OUT | THEY POINT TOWARD | | :------------------------------ | :---------------------------- | | Memory as more storage | Memory as control policy | | "Recall less" (the gate null) | Rank by the active sub-goal | | Finer atoms as the next lever | better ranking | | Passive retrieval | Memory authority | [Diagram showing "RANK" as a central concept, connected by arrows to four other concepts: "WRITE" (what becomes memory), "PRESERVE" (what survives eviction), "SUPERSEDE" (which fact still governs), and "INJECT" (how it's rendered). These four are labeled "four hypothesized".] ### 00:11:53 ## THE AGENDA # From storing memory to learning its authority **MEASUREMENTS RULE OUT** - Memory as more storage - "Recall less" (the gate null) ### 00:12:24 # AI Engineer World's Fair PRESENTED BY Microsoft ## Agentic Memory Solutions Lanscape - **Ledger / file** - In-house three-tier. ### 00:12:49 ### WHY LOCAL + A HARNESS ## Sovereign capability is a control board A local model plus a governed harness is the control board — capability you measure, not rent. **DATA & MODELS** You decide what data stays local and which open weights (GLM - DS4) you adapt and freeze. **COMPUTE & EVALUATION** One ### 00:13:20 ### WHY LOCAL + A HARNESS # Sovereign capability is a control board A local model plus a governed harness is the control board — **capability you measure, not rent.** ## DATA & MODELS ### 00:13:47 # Engineering the future of AI ## sakana.ai Building Frontier AI in Japan https://sakana.ai/careers/ [Screenshot of the sakana.ai website homepage, featuring ### 00:14:17 ## AI.ENGINEER # You can't prompt the room. Cheap to build. Expensive to decide. ### 00:14:49 # You can't prompt the room. Cheap to build. Expensive to decide. Balazs Horvath ### 00:15:21 ## WHERE THIS STARTS # 17 of 21 agents abandoned On data access. None of them reached the model. [Grid of 21 squares, ### 00:15:50 ### THE BRIDGE # I've always prompted the developers. ## The room was always the hard part. - 13 years, ERP & CRM - US · UK · Hungary - VisualLabs, Microsoft partner [Icon of an open book] ### 00:16:16 ### THE BRIDGE ## I've always prompted the developers. The room was always the hard part. **13 years, ERP & CRM** | **US · UK · Hungary** | **VisualLabs, Microsoft partner** [Icon of an open book] ### 00:16:49 ## THE SHIFT # The bottleneck moved. - USED TO BE - getting the code built - NOW - deciding what to build [Diagram showing an arrow pointing from "getting the code built" to "deciding what to build"] ### 00:17:19 - **prompt** your code ✔ - **prompt** your AI ✔ - **prompt** the whole spec ✔ # You can't prompt the **~~room~~**. ❌ ### 00:17:51 # WHY A MODEL CAN'T DO IT ## Ask it what to build. You get a faster horse. - what already exists - the average - a faster horse after Henry Ford. [Flow diagram showing three connected oval shapes: "what already exists" points to "the average", which then points to "a faster horse"] ### 00:18:25 ### WHY A MODEL CAN'T DO IT ## Ask it what to build. You get a faster horse. > after Henry Ford [Flow diagram showing "what already exists" leading to "the average", which then leads to "a faster horse"] ### 00:18:44 ### THE CRAFT COMES BACK ## The analyst's toolkit is senior work now. Story mapping is one example among several frameworks. ### 00:19:19 # THE STORY MAP - Contact - Triage - Resolve or route - Close ## RELEASE 1 - Capture intent - Classify urgency - Draft a grounded answer - Log to system of record ## LATER - Read sentiment - Route to a team - Suggest next action - Check satisfaction [Flow diagram showing a story map with main process steps and associated tasks grouped into "Release 1" and "Later"] ### 00:19:45 # THE STORY MAP - Contact - Triage - Resolve or route - Close ## RELEASE 1 - Capture intent - Classify urgency - Draft a grounded answer - Log to system of record ## LATER - Read sentiment - Route to a team - Suggest next action - Check satisfaction [A story map diagram illustrating a process flow. The top row shows four main steps: Contact, Triage, Resolve or route, and Close. Below these, two sections, "RELEASE 1" and "LATER", list sub-tasks associated with the main steps.]