# Frontier Models for Acting in Space and Time โ€” Pim de Witte โ€” session 2026-10-13T20:48:00.000Z โ†’ 2026-10-13T21:06:00.000Z _60 transcript lines ยท 36 slides_ ## Transcript So people use Metal to record themselves playing tens of thousands of different video games. And if Think about it, as a human, we can tell operator robots with the exact same brain that we played that video game. And in fact, in many cases, we also can use the same controller. So what this means is that the action space of video games, often because of game controllers, and the action space of teleoperations or the real world, in many cases is the same. And so every single robot, the Hello, certain degrees of freedom ships with the game controller. And majority of the supply chains already work this way. And so we thought, what if you just take all of the action data that we get through the video recorder, which associates the inputs and the actions with the actual frames. We label the actions, so we understand semantic meaning behind every action, and we use it to give it to a vision based model. And so what you see here on the left is just an intuitive way of how the model works. So you give the model a set Keys to choose from, essentially a computer use model. It gives frames as the input, so you can give it a text prompt as well. And you send it through API, and you say, somebody weighs it, you wave back. In the input space, the model understands that you have agreed action available. It will output the action associated with that. One way of intuitively understanding how this works, so, I've obviously simplified here because it's predicting the next token, not the next word, but bear with me here. LMs took large scale Internet data, then trained it to predict the next text token, and now use it for writing software. And these are all workloads that are really, really well fit for futures that are not simultaneously unfolding as you are predicting, right? Code is turn based. You can say something, the model will do something. You can say something again, the model will do something again. And, you can use it for lots of valuable work, but at the end of the day, because it's turn based, it's very, very Difficult for it to generalize to environments, especially that that include, space and time. And so our models, the way that they work is they're continuously receiving frames or receiving batches of, essentially, information from the camera of the robot, or it could be a screen. And it predicts everything as if it were just a video game. So you can think of the way that anthropic focused on just nailing Coding. So, you know, they were never frontier and just base text generation. They always focused on being the best at generating code. And then what did the bet that they made is that more people would start bringing more problems into coding space. If you think of general intuition, our bet is that you can, as humans, we are generally pixel to action converters plus senses, which in many cases, you can convert into pixels as well. And you can output actions at the same time. And those actions are, in this case, either keyboard and mouse or a game control. So the output tokens of the model are, essentially equivalent to that of enhanced computer use model with more more more actions on the output space to internalize it. So you send frames, and you get actions. This works, again, because every single thing that you see here can control all of these different embodiments, right? So, as humans, we do not know how to predict motor torque. We do not know What value to send to a machine? There's this beautiful device that has been made for many years, which actually does that work for us so we can control them. Most robotics companies today are trying to solve the problem at the end to end level, trying to predict exact poses or exact, values, in the motor, instead of just looking and picking up the thing that every robotics company has built to generalize our very Limited, abilities to have physics understanding and predictive text values, and solve for it. And so, our approach is, let's just use that. And so, Metal, at the moment, sees about a billion videos uploaded per year. This year, we are on track for 3,000,000,000, which lead to over a trillion and a half, action tokens per month. So just for comparison, to think about the bets. The total amount of data that Frontier Labs use today to train robotics models, we collect in, at the moment, half a day. And so, the the the type of bet... And again, think of the fact that humans can use the exact same controllers and their eyes to play video games that they can't tell to upgrade robots it is the same thing. We just bet on scaling pixel space the same way that anthropic scaled coding space and betting that people will just, as we scale data, compute. Into pixel space, so people will bring more problems into pixel space. And so what happens? All of a sudden, you get an incredibly general agent that can go and solve problems on computers, in the real world. And in addition, similarly, Talanthropic focuses on doing coding environments. We focus on world models, which most of you know us for The Mira release, and world models to then generate additional environments for these models to learn in. One way of internalizing a big advantage of this approach against LMs is if you're an if you're an LLM company and you have to bet on the ability to generate environments for your models to learn, you have you have two problems. One, you actually need to spend the compute to simulate environments, and the more realistic you try to simulate, the more expensive that gets. And so imagine you have... You want to learn a new thing. You want to learn That new thing in the exact input space of the human. Right? So you... If if we are learning from pixels, we're we're learning by looking around. If I'm trying to communicate something, I want my robot partner to observe that exact same thing. And so, what what I don't want is for that to have to go through an LM, for a digital twin to have to be created, for the model to have to get deployed, for it to have to be a bunch of verifier checks whether or not things are equivalent, and then for the robot to to learn much later. Right? And so One way of thinking about it is when you can learn in the native modality of the human, you can all of the sudden start solving a lot more problems a lot faster. What you'll see when these models come out is they are significantly faster than any L. M. At doing these things. Because games are so action dense, so what you see, right, is, one thing that is interesting is, you saw, for instance, the model predicting multiple actions at the same time. We are able To take that action density from gamers and convert them into really complex robotics workloads. So for example, if you are operating a drone, and you need to make lots of little adjustments, you really, really, really want to start with games. You do not want to start with teleoperations because the human will just take a lot less actions. It has a lot less information. So you want a model that can predict really high APM. And so the APM of the data set puts a lot of these problems all of a sudden in reach so that it can do a lot These actions simultaneously. And as I mentioned, the model doesn't just understand the key it's predicting. It also understands the semantic meaning associated with that key it's predicting, which is really important for things like, interpretability. So we will be releasing the models this year. We are very happy. I think the correct word would be that we are stunned by the fact of what these models can do. This is a live stream, so I unfortunately couldn't share the release videos without breaking the surprise. But if you're in New York, we are also in New York. And so, we have started doing a few demos, of the model. So if you can know know somebody on the team, reach out, and we can start showing you some of the capabilities live. But we have essentially one model, which can play video games at the highest level, and control over 10 types of robots, using the exact same primitives. So pixels in, actions out, completely real time. And, we'll be rolling out early access to these models over the next few months. We already have some customers that are on them, but it's still, private, so you're not hearing too much about it. But if you scan this QR code, you'll be able to get the models, because we go by who signs up on the wait list at which times. So, this is a great this is a great moment to get in touch. As I mentioned, we are in New York. I think what you will find is similar to how, These decision making models have kind of opened people's eyes on what you can do when you have a new architectural primitive. People are ready to kind of do away with the unpredictability of an L. M. And put something in that is more reliable and faster, and you just understand how it it is going to react, as opposed to having to work around it, if that resonates, these models are exactly what you are looking for. And all of a sudden, you can start exploring the environments that you generated with LLMs. All of a sudden, one of the Case that I'm really, really excited about is, well, these models, they, can take actions as if a human was on the interface. And so if you're doing UX work and you're trying to understand, is this an intuitive interface? Well, this is a very interesting way of solving the problem. So all these problems that LMs can't solve because they can't actually explore like a human, you can use these problems to start... You can start using these models for. So, the way that they're going to be released is we still expect you to use Use an orchestrator model. So you may use something like Gemini or Astra in order to do long term planning and and everything, and you'll be able to plug these into your workloads. And we're actually... The the models are so general. It it is genuinely going to, I think, feel like the early LM days, when you have a new toy, and you just want to figure out what it does, and all the things you can use it for. My reaction was like that when when I when I first started using the checkpoints, and I think I think you'll have the In reaction. And I think a lot of you will... Just like with LMS, if you're a bit early and you start understanding these primitives, I think you'll understand, that you can even build lots of businesses on top of this. Right? You know, think about, for instance, a science lab. Right? If you, all of a sudden, if you can reliably orchestrate robots in space and time, and they can take text instructions and do things, all of a sudden, This is a recursive problem. Right? Robotics is interesting because one of the big problems of solving robotics is making more robots. And so there are all these businesses that, because it's so bottlenecked by intelligence, are going to emerge just from solving this one simple problem, which is pixels to actions. And I think if you are early on getting your hands on these, it will have a similar effect on getting your hands on early LLMs. I am most excited also One of the things that that we are working on, which is interesting, is if you have any friends or family who, are, in some way, need accessibility help for using their computer, because these models can use computers like a human, all of a sudden, for instance, if you are if you are, a view only, you can use one arm. All of a sudden, the model can allow you to play video games because you can you can use... You can just control the mouse, and the model is predicting the actions for your keyboard. And because the model also understands which Actions you're predicting because that's the input. Right? It's it's observing as it's acting. It will actually just not predict the things that you're doing, and it will just predict the keys. And so it has all these downstream effects of things that you can build on top that I'm an engineer. If you're interested in building in this space, this is going to be the most interesting toy that comes out this year. I, I am fairly certain. And I also think a lot of the initial usage will sit in simulation. So if a lot of you are into generating With Astra or Gemini is perfect. Now all of a sudden, you can have something that can explore it, and you don't have to do it yourself. And it's perfect timing because of all the environment generation capabilities going going into these models. Okay. So really quickly before I wrap up, I want to talk about the team. So we are based in New York City. The Metal team and the general intuition team is here in New York City. So if you're here, you should reach out to us. And Can both show you the models, especially if you want to join the team. I think we will prioritize even higher. But, we really are looking for, people who, A, want to understand these models as much as we do. So people are... If you're really, really good at generating novel environments, typically, if you have some sort of research background, but I don't have a research background, so that's okay. If you, you know, if you, if you come from an engineering background and you're very, very happy to learn, this is this is fine too. And ideally What we want is somebody who, has experienced shipping end to end, AI products, but it doesn't have to be on the research side. So we just want to kind of hire people that understand the entire thing. Let's see. We have three teams. So one is world modeling. The other is the policy, so the agent. And the other one is working on, essentially, the data generation and the environment generation. So these are the main teams. If you're an engineer, the best one to join is The environment generation team, and then you sort of grow into, getting access either once. Yeah, we... I'm very, very excited for November. I really hope that you'll be on the lookout. As I mentioned, it will be either late November or early December is the current planning. One more thing, we are going to organize three hackathons. So the first is going to be in New York City. It will be mid November. And, we will send out If you sign up for the partner wait list, we'll send out some emails so that everybody can get access. We hope to see everybody at the hackathons. I think it will be really the first time to get your hands on them, the way that we're playing with them internally now. And, there will be one in New York City and one in London. So if you're in either of the cities or want to travel to either of those cities, those would be good times to meet the team. Yeah, let's see. I have time for one or two questions, and then, I think we can wrap. Yeah. How is it different from JEPA? So JEPA does not train alongside the action labels, so they bet on inferred action labels, for the most part, which I think... One of the ways of thinking about it is JEPA, I think, in the limit is the more correct approach, but it is just a lot less practical in the short term, because you need to bet on a separate skilling law on the action inference itself. And, we really Like Jetpack as a concept, we think it is a, you know, we we use some of the things in the models. And, but at the end of the day, it is an approach, not a product. And I think we will, you know, borrow concepts both ways. One way of thinking about it is, because we have so much action data let's say you're flying a plane, and you're moving the rudder in the back. There's no way of being able to tell that In the video frames. It's it's very, very difficult. So our mode on having the action data really transfers better to action sequences than it does to world models and to to generating the pixels. And that's why we just went all in on training these general foundation models for action predictions. And we use all sorts of techniques to do world models. You know, our own, we really like other companies building them as well. So great. Move one more, and then I gotta run, I think. Great. Good. Alright. Thank you, everyone. Have a good one. Ladies and gentlemen, for our final talk of the day, please welcome to the stage the CEO of Nus Research, Dylan Rolnick. Hey, everyone. I'm Dylan Rolnick. I'm the CEO of News Research. And today, we're going to talk about building on foundations and platforms that you can actually own. Whoop, there we go. Alright. How do you make back a cent a billion dollar infrastructure and research bet? Labs individually have spent hundreds of billions of dollars training and building ever more sophisticated intelligence system, and the industry as a whole has spent trillions, perhaps. The problem is ## Slides ### 00:00:18 # The frontier lab for acting in space and time Truly intelligent machines must move from words to worlds, and acquire the capacity to perceive, anticipate, and improvise. They need to obtain a ### 00:00:56 ## Frontier Models for Acting in Space and Time ### The frontier lab for acting in space and time Truly intelligent machines must move from words to worlds, and acquire the capacity to perceive, anticipate, ### 00:01:16 # Our action models see the world, and predict actions Someone waves -> Action API -> "Wave back!" ### Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Diagram illustrating a process: a video frame showing a person waving, an arrow pointing to "Action API" with a logo, then another arrow pointing to a video frame of an empty room, labeled "Wave back!"] ### 00:01:50 # Frontier Models for Acting in Space and Time ## Our action models see the world, and predict actions - Someone waves - Action API - "Wave back!" Pim de ### 00:02:18 # LLMs predict the next word, our models predict the next action | | PRE-TRAINED ON | LEARNS TO PREDICT | NOW USED FOR | | :---------------- | :------------------------------------ | :------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### 00:02:50 # LLMs predict the next word, our models predict the next action ## Language models - **PRE-TRAINED ON**: Trillions of words from the internet - **LEARNS TO PREDICT**: The next word - **NOW USED FOR**: - Writing software - Research & analysis - Agent run workflows Same recipe, different data ## Action models - **PRE-TRAINED ON**: Millions of hours of action-labelled gameplay - **LEARNS TO PREDICT**: The next action: Keyboard strokes and ### 00:03:20 # LLMs predict the next word, our models predict the next action | | PRE-TRAINED ON | LEARNS TO PREDICT | NOW USED FOR | |---|---|---|---| ### 00:03:50 # LLMs predict the next word, our models predict the next action ## Language models ### PRE-TRAINED ON - Trillions of words from the internet ### LEARNS TO PRE ### 00:04:19 # Humans play video games and operate robots with same primitives Humans play video games and (tele)-operate robots using the same inputs and outputs, namely vision & controller actions ### Frontier Models for Acting ### 00:04:50 # Humans play video games and operate robots with same primitives Humans play video games and (tele)-operate robots using the same inputs and outputs, namely vision & controller actions [Illustration of a game controller connected by dashed lines to a monitor displaying a video game, a quadruped robot, a drone, and a wheeled robot] ### Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition ### 00:05:20 # Medal is currently the fastest growing company in games ## Clips uploaded per month - Cumulative uploads - Monthly uploads The total amount of data frontier robotics models are trained on today Medal ### 00:05:51 # A new class of foundation model emerges ## Our action models: - Capable of handling robust autonomy - ... from pixels to action - ... across a broad range of embodiments - ... across a broad range of environments - ... in real time - ... in games and in the real world MORE SOON [Image of a robotic arm with a gripper, moving in a room] ### 00:06:15 # A new class of foundation model emerges Our action models: - Capable of handling robust autonomy - ... from pixels to action - ... across a broad range of embodiments - ... across a broad range of environments - ... in real time - ... in games and in the real world MORE SOON ## Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Video placeholder with a play button icon] ### 00:06:51 # A new class of foundation model emerges ## Our action models: - Capable of handling robust autonomy - ... from pixels to action - ... across a broad range of embodiments - ... across a broad range of environments - ... in real time - ... in games and in the real world MORE SOON # Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Video placeholder with a play button] ### 00:07:17 # A new class of foundation model emerges Our action models: - Capable of handling robust autonomy - ... from pixels to action - ... across a broad range of embodiments - ... across a broad range of environments - ... in real time - ... in games and in the real world MORE SOON ### Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Video placeholder with a play button icon] ### 00:07:51 # Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO, general intuition ## The frontier lab for acting in space and time ### Get Early Access ### 00:08:22 # Get Early Access The frontier lab for acting in space and time [https://generalintuition.com/](https://generalintuition.com/) general intuition --- Front ### 00:08:51 # Frontier Models for Acting in Space and Time ## The frontier lab for acting in space and time ### Get Early Access [https://generalintuition.ai/early-access](https://generalintuition.ai/early ### 00:09:24 # Get Early Access The frontier lab for acting in space and time [https://generalintuition.ai/early-access](https://generalintuition.ai/early-access) ### 00:09:50 # The frontier lab for acting in space and time ## Get Early Access [https://generalintuition.ai/early-access](https://generalintuition.ai/early-access) ### 00:10:23 # Frontier Models for Acting in Space and Time The frontier lab for acting in space and time ## Get Early Access [https://generalintuition.ai/](https://generalintuition ### 00:10:50 # The frontier lab for acting in space and time ## Get Early Access [https://generalintuition.com/early-access](https://generalintuition.com/early-access ### 00:11:22 # Get Early Access The frontier lab for acting in space and time [https://generalintuition.com/early-access](https://generalintuition.com/early-access) general intuition ## Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO [QR code linking to generalintuition.com/early-access] [Abstract image resembling antlers or branches, rendered in ASCII art style] ### 00:11:43 # The frontier lab for acting in space and time ## Get Early Access [https://generalintuition.ai/early-access](https://generalintuition.ai/early-access) general intuition ### Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Abstract image of a form resembling an animal or plant, composed of small characters and pixels] ### 00:12:23 ### The frontier lab for acting in space and time # Get Early Access [https://generalintuition.com/early-access](https://generalintuition.com/early-access) ### 00:12:47 # Get Early Access ### The frontier lab for acting in space and time [https://generalintuition.ai/early-access](https://generalintuition.ai/early-access) ### 00:13:18 # The frontier lab for acting in space and time ## Get Early Access [https://generalintuition.ai/early-access](https://generalintuition.ai/early-access) general intuition [QR code for early access] [Abstract image of an animal head with antlers, rendered using text characters] ### 00:13:45 # Thank you! Stay in touch! pimdw@generalintuition.com We're hiring Scan for open roles. ## Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Group photo of people sitting on a couch with a robot dog in front] [QR code linking to https://generalintuition.com/careers] ### 00:14:23 # Thank you! Stay in touch! pimdw@generalintuition.com We're hiring Scan for open roles. [https://www.generalintuition.com/careers](https://www.generalintuition.com/careers) ## Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Photo of a team of about 15 people, including a small robot dog, sitting on and around a sofa] [QR code linking to generalintuition.com/careers] ### 00:14:47 # Thank you! Stay in touch! pimdw@generalintuition.com We're hiring Scan for open roles. [https://generalintuition.com/careers](https://generalintuition.com/careers) ### Frontier Models for Acting in Space and Time Pim de Witte / Co-Founder & CEO general intuition [Group photo of the General Intuition team, including a robot dog] [QR code linking to generalintuition.com/careers] ### 00:15:27 # Thank you! ## Stay in touch! pimdw@generalintuition.com ## We're hiring Scan for open roles. ### Frontier Models for Acting in Space and Time ### 00:15:48 ## Thank you! Stay in touch! pimdw@generalintuition.com We're hiring Scan for open roles. ### Frontier Models for Acting in Space and ### 00:16:19 # Thank you! Stay in touch! pimdw@generalintuition.com We're hiring Scan for open roles. ## Frontier Models for Acting in Space and ### 00:16:51 # Thank you! Stay in touch! pimdw@generalintuition.com We're hiring Scan for open roles. ### Frontier Models for Acting in Space and Time ### 00:17:15 # AI Engineer ## NEW YORK PRESENTED BY arize [Stylized graphic of an art deco city skyline] [Arize logo] ### 00:17:51 [Abstract geometric patterns resembling city architecture]