# Everything will be Simulated — Cristóbal Valenzuela — session 2026-10-14T21:06:00.000Z → 2026-10-14T21:24:00.000Z _67 transcript lines · 35 slides_ ## Transcript To see if the sub agent would change its behavior based on your Tweaks. And the reason to do this is obviously far worse than forking. But you do this because this is much cheaper, much faster, and allows you, as an Agent Builder, to really easily iterate on ideas without needing to go through the expensive process of actually forking an entire environment. And you can do this today without building anything. I did this for literally a year, year and a half plus, before we built forking. Into our internal systems. And the main idea here is understanding that, if you look at the distribution from earlier, you're trying to understand, is the is the agent generalizing to that distribution? And so you don't know purely based on the outcome eval. So you can try to choose the best outcome eval that represent the curve, but you don't really know. And so supervising the process through some of these methods gives you some additional signal in order to understand if it's general. Not going to be able to develop that signal. And now, if we go back to the firm, we talked about supervision. We talked about how the agent, if he's opinionated, can be more reliable and actually can be simple. What that means is the agent maybe it starts doing something narrow. But because people will work with it, they see that it's easy to use, and they see that it's reliable. As it gets better and better, they will trust it more. It's just like a coworker. You You have a coworker day one. Imagine someone sends you a PR. You're not going to if it's you're looking at two axes whenever someone sends you a PR. It's these two axes. What's the blast radius of the PR? And what is your perception of how competent that person is? Right? If the least competent person at your company is sending you a PR that's editing a core service, you're going to look at it. If the most competent person is sending you a PR that's editing an internal document, you may not look at it as much. And so this idea is true with humans working with agents, just like it is with humans working with humans. And I think I just want to make one last point to close here, which is that one of the things that people say when they hear this is, well, wait a second. If you let the agents just do what they want, won't they find A better way? Won't they move 37 your work? And the answer is, they might. But that's not how you scale GEP and scale output. Nobody wants their agent to move 37 their tax return. They want it done reliably, at scale, in a way that they can trust, and in a way that their people know how to work with. And so, the thing I will leave you with is, for agents to diffuse, they need to be simple. For teams to trust them, they need to be reliable. The way you get to simple and reliable is you develop opinions on how the Agent behaves. And then opinions allow you to build that additional signal that helps you start to close the loop, so that you can achieve self improving Agents in production. Thank you very much. What if AI could do more than generate content and Actually simulate the physical world. To explore the next frontier of world models and what they could unlock, please welcome Cristobal Valenzuela, cofounder and co CEO of Runway. Hey, guys. Very, very excited to be here. Thank you for staying for the last talk. Welcome to my talk about the matrix and how we're living through it. So, you know, I've I've been working on Runway for almost a decade now. Started here in New York, actually, maybe around ten years ago. And, I I want to walk you through some of the learnings we've seen through, you know, building the company, building this research lab here in New York, and where do we see a lot of the work that we do in video, in pixel generation really moving forward. And so I don't Kind of exaggerating. I joke that this is... Might be a simulation or not. And and so for maybe for perspective, I'll walk you through, a piece of history. This is like watching, prehistoric ancient ruins. This is our first AI film we ever made. It was done in 2019. It's called, The Orange, because as you can tell very clearly, it's an orange that falls from the tree and walks through different parts of the world. This is done. This is a video model, an image model that we Back then, that was creating pixels in a 128 pixels by a 128 pixels that would then forcibly up sample to to this kind of resolution. I think of this as art now. This is this is like looking at a film from the nineteen hundreds, you know, it's it's beautiful. There's a lot of history behind it. But this is 2019. We started the company in 2018, 2017. And so we're trying to figure out if this had any lags. We're trying to convince ourselves that if you scale this, it will naturally improve. To the point where we can get somewhere close to this. This is state of the art video today. This is all generated. None of these characters are real. None of those locations are real. I'm sure you've you've... If you're probably here, you know this, by now. You probably have seen this online, on Twitter. The models are just incredibly good. And the most interesting thing is that when I look at stuff like this, I still believe this will look like the orange in just a couple more years. And so I want to walk you through, you know, how we're thinking not only Through what we're doing at Runway today, but also in in the next couple of years and the research that's shaping that. We are a company that's at the intersection of both both art and science. We have deep roots in artistic and creative communities, and we have some of the best researchers in the world working on diffusion models and large video models. And I like that combination because it allows us to, you know, bring the best of both worlds. We're we're a team of scientists and artists thinking about the problem of Scaling video and world models. But in a way, the the core idea of the company was already discovered by cinema, and by moving pictures a hundred and twenty, a hundred and thirty years ago, which is if you think about video, and maybe you remember this from, from high school, video is an optical illusion. Video is the illusion of movement that if you do if you stitch frames fast enough, you will create movement. You will create the illusion of movement. You create motion pictures. And so at Runway, we're doing kind of something very similar. We're taking that idea really to the next level. How do we do next frame prediction fast enough, scale enough, in in in such a way that we can hopefully do more than just pretty pretty and nice creative videos? And so Allotus Allotus is powered by, of course, scaling laws. I think we believed in scaling laws, not only applied for language models, but also for video. And this has been true. We're now really ramping up the amount of compute that we put behind the the models that you already have seen. And and The the interesting thing about, you know, seeing the model scale over time is the emerging properties that appear, the intelligence that appears when you start taking this very simple idea that cinema found itself, figuring out a hundred and twenty years ago, which is next frame prediction, and taking next frame prediction really to the next level. Right? So if I take an image like this, which is not real, those two people are not real, and I just keep on predicting what the next frame would look like based on the previous frames, and if I do it fast enough If I do it with a network that's big enough, I'll get movement, I'll get deletion of movement. I'll get something that looks so real to us that we we humans can distinguish. I I look at that video and it's... I've been doing this for ten years. I can't tell you if that's real or not. And we measure that. We've actually measured and quant... Tried to quantify, if people can distinguish between real and general videos. And I think that that Turing task we already passed, people are not able to distinguish. But the more interesting thing is here is that really what's happening as you predict next frames You're learning a lot more than just pretty frames. You're learning about physics. You're learning about object permanence. You're learning about gravity. You're learning about occlusion. You're learning about movements. You're learning about how the objects work in the real world, And you're doing that all through just video, through images, through next frame prediction. So if you look at a complex thing like this, a good video model for it to understand how to make this a consistent video will need to naturally understand. And, what all these objects are, how each of them relate to one another, how a camel might probably, stand on a boat like that, what water in a container looks like. And if I do all of that together and I do it fast enough, I'll get, hopefully, a consistent video that looks real to us. So a lot of the benchmarks that we use to evaluate our video models are not only creative, it's creative basically means do you like it or do you don't, are actually based on, on physics. On geometries, on how objects behave in the real world and how those objects should behave, and also simulator or digital worlds. So you take that idea of an extreme prediction and you do it fast enough and you get, of course, now something that looks more like what we should expect from a video application or a game application. Again, this is just just video. Right? And you take you take that idea to a next level. What happens if we start this idea of an extra prediction but do it in the context of a three d environment or something that looks similar to a game? You can start We're getting closer to a physics engine, a simulation system. But the simulation system has no physics loss. We didn't describe, what the behavior of the object should be. The model just knows how to do that, by poor force of of, of scale. And this kind of gets gets kind of interesting because you can only think about playing Worlds. This is not GTA. This is, someone playing someone else, in a environment and a software that doesn't Yes. It's all simulation of a simulation, which which gets really meta. Right? And so I'm not kidding when I said welcome to my metric simulation. But the most interesting thing for me is that everything you saw before was not real time, meaning that it took a couple minutes or seconds for those videos to be made. A lot of what we're seeing right now when we think about, you know, the next three, four years of video models and world models is to deeply think about what happens when you start not only scale In the models and making them much, much, much, bigger, which much more emerging properties, but also what happens when you start making them interactive. Right? So take the same idea of next frame prediction. You take a frame, you need to predict the next frame, then x, then x, then x. But then it can act at any point in time in that frame prediction, which means I can simulate any possible divergence or action in that world. So in this world, I could play the video, but I can also come and touch every part, any aspect of that email. And make it interactive to affect the consequences of, of that world. And it works. And of course, this is incredibly useful for a variety of different use cases, as you can imagine. We're now spinning an entire robotics lab in Runway. So if you come to our office in New York, half of it is, what looks like a Hollywood studio. Half of it is a bunch of robots walking through and moving through the office. And so we're able to make these models or start make These models, the brains of how autonomous and embodiment systems walk and reason about the world. But there's a couple of more interesting use cases that I think go beyond the two obvious one, which is video and robotics, which is what happens when you think about anything, anything, any pixel on the screen as a video model. Here's a great example. This is a video model. What you see right now, well, this is an image, first of all. That image could be real or not, but I could run one of our video models in real time. Make this an interactive application. So let's say I wanna, you know, turn the lights of the car. Maybe I wanna like turn the car around. I can just click through it. I can turn it around again, maybe open the hood and see what's inside the car. Now everything you just saw here has no physical three d representation. There's no Unity. There's no Unreal. There's no Blender. It's just a video model. A video model has been trained to understand actions and clicks in an interface. We call There's a world action model. Sorry, a world interface model. And the world interface model, well, I I feel that you guys understand now. It it's it's an interface. And it's a world model, world interface model, allows you to do very cool stuff like this. And so here's another application, another example of how you could think about using a model like this in real time. I can take an image, a representation, move objects around, and expect them to move as I was... If if as I was actually in that room. Again, Remember, there's no three d representation of this. There... This environment might not actually be have... Have been modeled before. It's just an image and the model that understands how objects move in a space because of this very simple idea that next frame prediction scale well enough can become a world model or a simulation engine. So with the eight minutes I have left, I want to show you this idea taken to another extreme. We're we're thinking not only about what happens if you can control this cool Applications and these cool interfaces. But what happens if you keep extrapolating? And I think, by the way, extrapolating might be one of the most important skills these days in in AI, in in company building. What happens over the next six months, twelve months, eighteen months may be one of the most important things to to try to predict and understand. And so we ask ourselves that question a lot at Runway. What happens... What will happen if we keep scaling the models, if we keep adding more data, if we keep making them better simulation systems, if we make them Universal simulation systems that can predict anything on a screen, literally anything on a screen. So we've been working on this idea, that we call continuum. And continuum is, the the idea of real time video models or world models taken really to the next the next degree. This is an operating system. The idea here is that you might open your computer at some point in the near future, and what you see streaming to your screen might be coming from a video model. So I'll show you next specific how this idea could could shape. This is Still very early research, but I want to show you this idea of combining video models as, as part of your operating system, as part of your daily work. So here's an example. I was researching, you know, San Francisco and the fog in San Francisco. I wanted to learn more about when and how it happens. This is structural data. This is actually a website. This is something, you can take from, from the internet. And now Contium has this idea that I can select any part of content in that website and make it a portal. We call this a portal. And the portal is a real time video. It's fully interactive and fully based on the content you're experiencing reading through. I probably won't read through all of that text. It's too much text for me. So I just wanna make a video, and I can make a video that is represented by by the text you see there. The... These portals can be three d visualizations. It can be any content that you're watching or interacting with. On the right, you can have all the visualizations and portals you've you've experienced before, anything you've been trying to learn. Basically, it's a it's a memory. It's just the disk of the imagination. Or the visualizations you've seen before. So here I was researching about batteries, and so I can go back to, the portal I had around batteries before. And since our real time interactive applications, I can click them anywhere. I can move them around. I can start asking questions, and the videos and and the content itself will change based on based on my interactions and the way I'm I'm I'm I'm managing through it. Here's another very cool example that when... The first time I saw it, when the team showed Was, like, blew my mind. Since it's a real time video, you can make it fully interactive with anything, not only with clicks, but, like, resizing the applications should also change the behavior. Right? So I can resize. I'm I'm learning about photosynthesis here. If I resize the application, I get a different experience. And since this is, again, full interactive, applications, I can click through it, navigate through parts of the cell or how photosynthesis work to get a better understanding of what I'm trying to learn and do. Again, that the one Thing that still feels to me, kind of uncertain about this is, I don't know how to call this. I don't know if this is... It doesn't feel like a visualization. It doesn't feel like a video. It doesn't feel like a game. It's something in between, and I think we need to do the collective work of figuring out how we call these kind of interactions in the first place. And I'm showing you here kind of the other thing was that you can change the structure of the prompt or what's actually creating that interaction in the first place. Here is another example. I'm gonna type something. And this is one of the ones that I feel has been the most underexplored yet. This is, we call this visual thinking. So we've connected continuum to a coding agent. And what the coding agent can do is it can use continuum to visualize your problem. So imagine you ask your coding agent to create a three d model for your bike. And instead of the model trying to just think through that via code, the model can spin a version of continuum to ## Slides ### 00:00:07 # Simulations ## Example 1. Render the agent's prompt, skills and tool descriptions into files. 2. A coding agent spawns subagents. 3. ### 00:00:33 # Simulations ## Example 1. Render the agent's prompt, skills and tool descriptions into files. 2. A coding agent spawns subagents. 3. Each reads ### 00:01:02 # Is it generalizing? - Behavior specs check it on runs, even without ground truth. - It shows you where the behavior holds, and where it doesn't. Process ### 00:01:44 # Opinions close the loop ## Signal - Outcome evals - A few cases, one score at the end. - Which step, which behavior? - Behavior specs - Every run, in evals and in production. written down ## Opinions A reward function, one level up. Forks and simulations: test the fix ## Improve the system the harness, sometimes the opinion ## The agent works in evals and in production Agents can run more of this loop, if they know what better means. --- ### Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis [Diagram illustrating a feedback loop for improving an AI system, connecting Signal, Opinions, a Reward Function, System Improvement, and Agent Work.] ### 00:02:05 # Back to the firm > The head of the firm trusts agent work the way they trust their people's. ## Process Supervision for Agents That Do Long-Horizon Work Mitchell ### 00:03:01 # Won't opinions stop it *finding a better way*? - We're not paid to **Move 37** someone's close. - We're ### 00:03:31 # If you take anything away: 1. For agents to diffuse, they need to be **simple**. 2. For teams to trust them, they need to be **reliable**. 3. Simple and reliable come from **opinions** about how the work gets done. 4. Opinions let you build **additional signal** that helps close the loop on self-improving agents in production. ## Process Supervision for Agents That Do Long-Horizon Work Mitchell Troyanovsky / Co-founder Basis ### 00:04:03 # AI Engineer NEW YORK PRESENTED BY arize [Stylized graphic of a cityscape with skyscrapers] ### 00:04:30 # runway ## Everything will be simulated Cristóbal Valenzuela ### 00:05:04 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Image of a tree with orange fruits on a green grassy hill, with one fruit on the ground] ### 00:05:30 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Close-up image of a shiny, orange, somewhat irregularly shaped object] ### 00:06:03 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Image of a man behind a counter in a small store, with stacks of canned goods and sausages on display.] ### 00:06:30 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Grid of black and white photographs showing a ballerina in various poses, depicting a sequence of movement] ### 00:07:03 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Grid of 24 black and white images showing a ballerina in various poses, some with motion blur] ### 00:07:32 # Everything will be Simulated ### Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Grid of black and white photographic studies showing a ballerina in various poses, some blurred to convey motion] ### 00:08:05 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Image of a woman and a man facing each other in a studio setting] ### 00:08:31 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Video showing a person's legs and feet in brown boots walking on a white surface] ### 00:09:03 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Diagram showing a wireframe globe at the center, connected by lines to six rectangular cards. Each card contains an abstract visual: concentric ellipses, wavy lines, flying bird shapes, wavy lines with dots, dots above and below a dashed line, and an inverted V-shape with dots. A small image of two people is visible inside the central globe.] ### 00:09:34 ## Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Diagram showing a camel with various objects (cup, bottle, bowls) stacked on its back, standing on a cushion. Lines connect different parts of this composite image to a vertical sequence of smaller cropped images on the right, each isolating a component, with corresponding dots and directional arrows.] ### 00:10:04 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Diagram showing a stack of several identical grey, low-polygon 3D landscape models (mountains, roads, structures) on the left, with an arrow pointing to a single instance of the same model on the right.] ### 00:10:32 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [3D rendered street scene with blocky buildings, trees, people walking on sidewalks, and birds on a power line] ### 00:11:02 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Image of a Rube Goldberg-like machine on a desk, with a bucket, wooden ramp, and a watering can positioned to water a small potted plant] ### 00:11:30 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Image of a Rube Goldberg machine on a desk, featuring a bucket, pulley, wooden ramp, small car, watering can, and a potted plant] ### 00:12:04 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [A robotic arm with a three-fingered gripper reaching towards a Rubik's cube on a wooden table, in what appears to be a lab setting.] ### 00:12:32 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [A pair of robotic arms positioned on a table, facing a Rubik's cube that rests on a small wooden stand.] ### 00:13:04 ### AI Engineer NEW YORK Presented by Arize # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Red car with headlights on, parked in an alleyway] ### 00:13:30 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Image of a car engine bay with orange high-voltage cabling highlighted and labeled "High-Voltage Cabling"] ### 00:14:03 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Image of a simulated living room with a blue sofa, coffee table, and plant, featuring an interactive color picker overlay showing different material options for the sofa.] ### 00:14:32 ## Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Diagram showing a 2x2 grid of teal circles connected to a purple square, which is then connected to an orange circle, with a mouse cursor pointing at the orange circle.] ### 00:15:03 # Continuum ## Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [The word "Continuum" with the letters "co" stylized as a landscape featuring hills, a river, and a cloud.] ### 00:15:34 ### Screenshot of a generative AI interface [Screenshot of a web-based interface with an input field labeled "Ask or do anything"] [A column of small thumbnail images on the right ### 00:16:04 ### Screenshot of NASA Earth Observatory article: Another Foggy Day in San Franciscans, particularly those who live near the Golden Gate Bridge. # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO **runway** [Image of the Golden Gate Bridge partially obscured by fog] ### 00:16:33 # Everything will be Simulated Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Screenshot of a Google search results page for "how do batteries work", showing ### 00:17:05 # Everything will be Simulated ## Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Screenshot of an application titled "Inside a Leaf" displaying an illustration of multiple green chloroplast ### 00:17:31 # Everything will be Simulated ## Inside a Leaf An airbrushed biology-textbook cutaway interface: a green leaf sliced open in a big block showing its cell layers, chloroplasts glowing inside the palisade cells, a stoma on the underside, sunlight rays entering from above, and small hand-lettered labels "Chloroplast", "Stoma", "CO₂ in", "O₂ out", with a small drawn sun dial at "Noon". Cristóbal Valenzuela / Co-Founder & Co-CEO runway [Illustration of a cross-section of a green leaf showing its internal cell structure]