Yeah. Thank you, Michael. I'm Jack McDonald. I am a principal at Apollo and COO of Redding Ridge Capital, which is our structured credit and CLO manager that is captive to Apollo. We're a separate, fully separate entity operating company that manages about 50,000,000,000 in assets and, all across kind of the structured credit ecosystem, primarily CLOs. I've been at Apollo since 2020, where various different Hats throughout the, my time there, across our financial institutions group, our asset backed group, and now, currently at Redding. So, yeah, happy to be here. -Awesome. So something that we've noticed in talking to a huge number of financial firms is that there's a pretty wide gap between when AI is nice to have and when AI can actually become mission critical to a business, and I think in Apollo's case, we're seeing that AI can really be transformed In how the firm operates. I'd love to hear some examples from you on the kinds of work your part of the business was doing, what made it complex, and what you're seeing now with AI. Yeah. So at Apollo, and specifically Reading Ridge, you know, we manage many different fund entities, investments. We're getting thousands and thousands of data points on a, like, daily basis coming in across all of our investments. We manage probably, you know, thousand named loan pools and have Credits across multiple different businesses that we're assessing on a daily basis and investing in. CLO management is very active, and it's traded in the broadly syndicated loan market. So having all that information at your fingertips is very, very important. So at Apollo, what we're doing is... And at Redding specifically is trying to enhance and ensure that all of our data and our data quality is accurate, consistent, and in a format that we can utilize. And make us much more practical and thoughtful when we're making decisions in a much fast paced environment. So all of this information is privatized and is not very, what I'll say, widely in the marketplace, like how public equities or public fixed income trades. It's very, fragmented in various documents. So we've looked to companies like GrowSpace to partner with, to come in and actually sit with our teams, work with us. And understand the actual calculations and information that we're looking to, and what's actually the golden source of truth, and then continuing to enhance the AI engines to understand that and repeat that, and just make us that much more efficient. -I think we have a bunch of engineers in the audience, and we have spoken a lot to you and I on fire drills that the firm does that used to take a team working a week, if not more. Can you make it concrete for us and maybe walk us through an example? Of something that was painful, time consuming, hard to do before, that now you can do more easily with road space and AI? Yeah, happy to. And I'll give kind of two examples that we, that I like to look to. The first being is just on the portfolio construction side. So CLOs are very test specific. They have a bunch of... They're very rules based. They have, like, a very specific waterfall of where money flows. So those all live in deep indenture documents and multiple different legal Agreements that... So, sitting from, like, a portfolio manager's perspective in portfolio construction, they are reviewing those documents, building Excel based models, and building mock portfolios with, call it, anywhere from 200 to 400 single credit names to create a portfolio, which would typically take a PM before, kind of, adopting row space, anywhere from two to three days in a manual Excel. Today, we're able to kind of get the latest and greatest information that's real time. Create a portfolio that would take somebody days to now do it in the matter of minutes. And then that allows these PMs to kind of build something to call it 80, 90% completion, and then kind of use their judgment and their thought to further refine that portfolio, that last 10%, to make it theirs. That's one of the most tangible examples that's kind of pulling in various forms of data and, you know, information from across multiple different things, whether it's in doc libraries. You know, our trade, and compliance systems, and real time pricing, and stuff like that. The other one that I think I would look to is our, kind of like, warehouse purchase activity. So, how we run our our business operates with having a bunch of bank financing, and we can go out and buy loans in the fixed income market at any given point. Historically, we would kind of do a trade rec and trade activity on a weekly basis that would go around of, here's your sale purchase. Activity from the week. Here's what we bought. Here's what we sold. And it was very manual. We were going into various, like, literally going into bank accounts to understand, statements, where things actually settled. And it was an individual team that was going and wrecking that out and then sending a reported email to the management team to say, here's the recap. Today, what we're doing now is we've been able to take that exact same report, drop it into row space from a visualization perspective. That report was Then created and now repeatable in the span of a minute. And now, all of that is automated. And then, from our team's perspective, we are just QA ing that and checking it each day to make sure that that looks right. And now, it's a much more rinse and routine type process, which historically has taken many hours of the week for two to three individuals on our team to actually wreck that out. Yeah. I love both of those those examples. So, you know, you aren't just reading the output of a chatbot to be informed. I think your team are making trading decisions, performing management decisions, so accuracy and quality are high. This is a two part question. I think a lot of people will ask us or ask generally, why can't you just plug this stuff into Cloud and or track your team and see what happens? And for someone like you, how does your team get the confidence in a product like RealSpace or other AI tools to say, we are going to make business decisions on this, make trades, All of that. How do you get to that level of trust? Yeah. I would say, Claude and the open AI's of the world, and like the chat GPT's, all the LLM's are, very beneficial, but it's all general information that that's being worked off of, and none of your own actual data. So it's great for research, or kind of building new ideas, or getting up to speed on some type of product or investment quite quickly. Where Rowspace is different in what we've kind of taken... And our approach to this... The the AI build out at Redding has been around owning our own data and having some type of platform and technology that sits on top of that, that we know is to be true. So the Rose Space team, which has been much different from what we've seen from other technology companies that we've worked with, is they're boots on the ground. So, like, Michael sits with our team on a daily basis, learning and understanding our information flow, because for one piece of data, that Might come from 20 different sources, and there's only one that's actually right. So the row space team is spending the time with our portfolio management team, with our finance teams, to understand the calculations, understand what data is actually correct, and override other sources of information, and kind of aggregate that. So to that end, we're able to kind of have trust in the system and, you know, appreciate that, okay, we've had our team go in and validate that this information is correct, and we can rely Eye on it, and it's coming from all of our... Whether it's our trade order management system, whether it's from our performance system, whether it's from Bloomberg, or any pricing index that we're pulling in, and know that it's accurate. Whereas, in a lot of other, you know, call it systems throughout even our own company, it's very fragmented, and we have to kind of lift one piece of information from over here and bring it back over here and kind of pull that together manually, whereas, GrowSpace is now just automating that all for us. Glad to hear that. At the firm at a higher level, it seemed to me for a long time, even early last year before cloud code became a thing, when Kerstor was still in the attack computation era, that Apollo was among the most AI pilled of the firms we spoke to, and that was everything from the firm's view on how to invest in an AI world, but operationally, we've been talking to you a long time ago, and you are in the we must rebuild the firm. In an AI native way, must have a platform, not point solutions. So I'm just curious, where did that come from, and what is it about Apollo's culture that made you so AI pilled, even before things took off? Yeah. So Apollo has always prided itself in being kind of the leader of the alternative asset management space and the financial services industry broadly. We pride ourselves in being very innovative and being at the forefront of all things change. And what that looks like. So I think, from the onset and the way our management team approaches how we operate, is always being up to skill and enhancing our team's capabilities and being able to be nimble and adapt to where the market's going. I think we appreciated that the AI world is here, and we need to adapt and adopt it, not, you know, discredit it or not utilize it. So we've been very forward and, you know, open minded about adopting new technologies and bringing those into our workflows and how we can More efficient and free up capacity for people to be more thoughtful and think and, you know, replace manual processes. So I think, at the core of its level, we've always been a very innovative firm that's very flat and open culture to be open to those types of change? And then... I'm blanking on the second part of your question. Sorry. Well, I guess so am I. One, maybe we could keep on going. So I think one thing that we've been thinking about is software engineering that has completely changed as a discipline. Like, the world is completely different if you are a technology company like like Rosebase, and maybe arguably law and customer service are almost there. Where do you think finance is on the spectrum of AI is here, it's made a difference, or we haven't even, you know, gotten to the bottom of the first inning? And what do you think is two part question again? The second part. If we're early on, what's necessary to really change the shape of finance with AI? Yeah, I think that's a great question. So I think finance broadly, these banks and large organizations are massive. They have a lot of, you know, back end, what I'll call middle back office support. And I think there's a lot of room to grow and enhance there. I think, probably, as you appreciate now, of working with us, that we are extremely complicated and have a lot of complexity in everything that we do. So it's not Really straightforward. I think, in finance broadly, especially, in call it the asset management, investment banking, the bank institution... Institutional world, I still think it's very early. I think there's some value for some of the, like, you know, model creation or getting up to speed on some company quite quickly and pulling and aggregating that type of research. But I think that's lower value add, at least, of where we stand today, and that's going to evolve. I think, like, you know, there's probably a need where there's less junior support over time for certain functions like that. But I think from where we are today and what we see from from, at least, my perspective at Reading, I think we're just trying to adopt this technology and make it, you know, as efficient as possible for, you know, routine tasks that frees up capacity for, you know, other projects because we kind of have, like, endless things to do and enhance at our company. And that's just giving us a lot more flexibility to Adopt that. Yeah. Yeah. So maybe continue on that note. I'm curious, there's a world in which AI just becomes something that helps speed up tasks, saves you on OPEX, things like that, And maybe in the same way, no given firm now has an advantage because they use Excel. Everyone uses Excel, and it's just a productivity tool. There's also a world in which AI actually materially makes a competitive advantage or helps reify Apollo's Lead in structuring and creative deals and so forth. Do you have a take on whether AI is going to be just a useful, but a general commodity or it's gonna be something that can actually structurally make a business more competitive in a durable way? I think if it's adapted and utilized properly, it will make you more competitive. I think at its core, and especially around asset management and investing type companies that are in the actively traded markets, like It comes down to your process and the information at your fingertips. So knowing that you have a strong process in place to go out and execute, and knowing your available liquidity or, you know, where things are pricing and what the value is at any given point is crucial. So that just gives you, you know, more competitive edge. And I think, like, that exists today in the, you know, actively traded markets. But in the private markets, it's evolving, which we've kind of been at the forefront of trying to move on the broad Apollo machine. And I think Having that will just turn into a competitive advantage over time, where firms will be able to get further ahead as they're, you know, having access to better information. They they know... Their teams know that it's accurate and, like, stuff is not misaligned. Because the last thing we wanna do is, you know, do something or make a decision that was misinformed or had misinformation, and then that streamlines down the chain to both the enterprise and to the... You know, our investor base at the end of the day. Yeah. That that makes sense. So I wanted to ask you what your prediction was for what Apollo would be like in... When we're talking five years observing all the change. I think that's unfair. I can't even predict if my team's gonna use code x or Apollo code next week, so I want to ask you about five years in the future. But I'm curious for you personally, what are you hoping to see in terms of your job, team, how AIU Sao Paulo evolves, what conditions have to hold for To actually mutually valuable. You know, what are you... How are you thinking about things over the next, you know, next few months? Yeah. I think it comes back to some of the stuff I touched on before of just, one, having just, you know, full confidence in any type of information that's flowing out at the end of the day to our investor base that's accurate, correct, and we don't have to spend hours manually checking that to make sure that that's validated and having the confidence that we're sharing, you know, no issues or mistakes, because we do have just a wide Investor base, you know, over 400 global investors. I would also say being much more tech enabled in mindset, I think our team has not lacked that to date, but it's kind of been an afterthought around the use of technology. And I think the more that we can develop and enhance these tools that make, you know, routine tasks, you know, more efficient, it frees up capacity to add more thought or think more creatively around why does this process exist and how does it Work today, and how can we actually enhance that? Or should we change that? Or should we bring in a new type of technology and adopt that going forward? And our team has not lacked that today, but it's kind of been a secondary thought, where I think, ideally, our workforce is positioned to think about that more on a go forward basis and that's kind of their primary thought is they're tackling different problems, specifically, this is specifically at writing Ridge day to day and kind of scaling our organization. I think we have time for one more. I think one of my learnings Working with you has just been how unimaginably complex and creative the work is, and how it takes a lot of effort, both human and AI, to get things to work. A lot of people in this room who either want to work on the technology side of AI and finance, or perhaps finance side, is there anything that you would mention to them from, here's something that you should know or keep in mind when trying to apply AI to domains like yours? Yeah. I would say it comes down to, we are extremely complex organizations, and it's not Straightforward and, typically, things might look a little bit novel or strange and out of place, but it's probably actually there for a reason or structured in a specific way. I'm sure as you guys have come in, you appreciate that we're super nuanced and complicated. So spending the time and really putting boots on the ground to sit with whatever team or front office professional that you're working to build a tool for or enhance is time well spent. And I think the success and, you know Where we are today with the row space products and kind of being ahead of our roadmap and expectations has been, you know, due to the fact that Michael and the team have spent ample time sitting on the ground with our team, like, pretty much every single day, and, you know, understanding our workflows, which I think is is crucial. -Awesome. Well, I guess last thing I'll say is that, shameless plug, both Rosebase and Apollo Technology are hiring people who want to apply AI to finance, so please reach out to either of us if you are interested. That's it. Jack, thanks so much, and Yeah. Great chatting. Thank you. This is great. Thank you, guys. When AI agents tackle complex tasks, the final answer is only part of the story. To explore how we can evaluate the decisions they make along the way, please welcome Mitchell Troianovsky, cofounder. Of Basis. Hello. Hello. Hello, New York. I'm Mitch. I'm one of the two co founders of Basis. We are based right here in New York City. And the thing that we We focus on is, how can we get agents to do production scale work reliably enough so that we are trusted and used end to end by accounting firms across the country. So a lot of our research really focuses on this. How can you get these agents to do real work in the real economy? So I wanted to start with a question, which is, if you're at this conference honestly, if you're watching This talk, you probably increased your own productivity a lot. You're using co work, or, codex, or cloud code, whatever it is. And yet, if you go look at the average company, the average enterprise, the average firm, their productivity hasn't gone up 10x, 100x. I know my productivity has gone up maybe 100x. But if I were working inside of a company, it wouldn't be anywhere near that order of magnitude. And so why is that? At Basis, one of the things we have is that humans are already used to working with non deterministic systems. It's just those systems are normally their coworkers, not their computers. And so I actually think, to answer this question, it's useful to maybe look at the existing way we organize intelligent systems, which are companies. So how do organizations scale their own productivity? Right? Organizations need to To trust that the people doing the work at every level are doing it in a way that is going to produce good outputs consistently at scale. Right? If you look at this image here, here I'm showing an accounting firm. But this could honestly be any company, traditional company. The person who is the head of the firm, they have literally no idea about the exact day to day of the people at the very bottom. And yet, somehow, they trust the work is going to be done. How does that happen? It does not happen through deterministic systems. Right? People are not deterministic. They are intelligent entities. And instead, there are processes that are put in place. And those processes, if we believe the people who, you know, put them in place think that those processes will generalize to good outcomes, then they can be comfortable that even though the people executing the processes are not deterministic, you will generally get to good outcomes if you follow the process. This is true, by the way, even The smallest step. So in this example, earlier I was showing the process of maybe doing the monthly accounting work for a client at an accounting firm. But even within that process say you wanted to see, how do I reconcile the cash? Well, I need to get the bank data. And even that has a process. The process could be really basic. It could just be, use the approved software to extract the data. But there's still some process that is put in place. You wouldn't expect the firm to just let anyone goog Cool, you know, extract PDF, and then just use whatever they want to go extract it. Because now, you would not have process. Now let us talk about agents. As I said earlier, everyone already has co work, ChatGPT, Codex, Cloud Code but where is that productivity? Our opinion is, there are two main problems that come with what I will call coding style Agent products, where you are working with an agent You just tell it to go and do something. The first problem is that there is a competency gap. Because agents, as you think of them as you think of them today, they outsource the complexity to the user. Right? If I am the user, I now have to manage my skill and memories, and think about how to, when I should use the Agent, I have to have like, theory of mind over it. This is really hard. And for those who are here, maybe you guys. Find it easy. But I promise, for the wide majority of knowledge workers, they have no idea how to do any of this. And there is a second problem, which is, there is a process gap. Right? If you are saying, hey, user, here's your Agent, do what you want, well, now, suddenly, the way the work is being done is essentially up to whoever the user is to customize the way the Agent is behaving, which goes entirely against the way that Organizations scale. The process should be coming from senior leadership down, not from any individual employee deciding how they're going to do their job. Right? If you going back to my example if every employee just made up their own way of extracting data, you know, even back in 2018, that would not be a good outcome. And so we believe strongly that the key to building production agents that actually do work end to end is solving these two problems. You have to solve the competency gap by making It's simple. To the user, you shouldn't even have to know it's an agent. Or you probably know, but you don't have to really feel it. It should be as simple as just working with a coworker. You just tell it what you want it to do, and all the things you expect to work should work. And the second is the process gap. You need agents to be reliable not having a stroke of genius one day and forgetting something the other day. It's not about being perfect. Humans are not perfect. Right? Nobody is looking for Perfect. What you are looking for is reliability and consistency and trustworthiness. So what does that actually look like? We think that to be simple and reliable, where you know you do not have to think about what to do the agent can just do it for you without the user knowing the internals to do it the same way, consistently, you have to be opinionated. And just to show you, actually, one slide back here. I want to point out that getting it right is not actually the main task. Because again, humans are imperfect themselves. The point is to do it in the similar way. It is okay if you can't do something. But you should be consistently able to not do something, and then you get better, just like a person does. If you suddenly, one day, are an expert at something, and then you miss something very basic that would be expected, now you are not reliable. So you have to be opinionated. What's an example of an opinion? Let's say we have an Agent, and we need to go and read a PDF. So, in this case, what is the best way to read a PDF? There's many ways. You can throw it into a VLM, You can go write some Python code and use an open source OCR library. You can work with one of the many OCR specific model providers. You could do an OCR, and then you could have the VLM check its work. You could have five VLMs check its work. You could do a 100 VLMs. And take the best of them. There's literally infinite possibilities here. And so, if you just said, Agent, extract Excel extract PDF, you might get good outcomes, but they're not going to be reliable. Right? Not because the process it did was bad, but because the process it did was not opinionated. So the way we develop opinions is, you use initial intuition. Write some things, you know what the opinions are from expertise. For example, if I'm trying to get an agent to do tax work well, I know that if the agent is going to goes and reads the primary source before returning to the user, it is more likely to be accurate. I don't need a set of evals to tell me that. I don't need to divine that from first principles. We know that. Right? I know that if I have an agent make a PowerPoint, if it takes a picture of the PowerPoint before it returns to the user, it is less like To have a formatting issue that is obvious. And so, there are lots of different ways to develop opinions. And one of the key points here I'll just make, before I move on to talking about how to use the opinions, is that the opinions are not just about accuracy. Right? We just talked about the example of taking a picture of a PowerPoint. Well, that adds latency. Right? You may not want that. Maybe the Agent only makes formatting mistakes one out of a 100 times, and you are okay with that. And so therefore, you do not have to take a picture of the PowerPoint. The point It is not exactly what the opinion is. And this talk, we are going to focus not on how to develop opinions, but why opinions matter, and then how to leverage those opinions as you scale in production. So say you have a bunch of outcome based Evals. And when I say outcome based Evals, I mean what we traditionally think of as Evals. It has a right answer. We have some rubric. We're trying to judge what the agent did. How do you know that the Vows generalize to production. Right? You might have 1,000 cases, or 100 cases. But in production, if you are doing real work, you are going to have hundreds of thousands of situations. And that is only going to increase as the trajectories get much longer, because the amount of decisions that are made in a trajectory just grow exponentially as the long horizon abilities of agents increase. And so, I can't prove, if I if you're looking Distribution of, trajectories. You can't prove every one of them is correct. You don't have a ground truth for every trajectory. But you can potentially look at the trajectories to decide if the behavior, the opinion that you have, is actually happening. And you can then infer that if it's actually happening, it's more likely to generalize to production. It's not perfect. You don't know for sure. But at least you are having an opinion. Let's run through a real example. So let's say that I, for a behavior this is an accounting example where if there is a mistake in the bank reconciliation, I don't want the agent to, like, post an entry to make a tie. I want it to sort of explain the difference, instead of putting a plug. Again, this is subjective. You might want the other way. But what you don't want is a customer getting one way One time, and another way the other time. And so, what would you do? Well, first, let's understand what governs the behavior. Right? The behavior is not just, you write the two sentences into your system prompt and it just works. That might work for very simple, narrow agents. But when you're building these big agent systems, the behavior is actually a combination of the system prompt, your progressive disclosures, potentially tool descriptions, obviously, the training of the model. There's a lot of things that govern the behavior. And one tip, by the way I have for when you think about building production agents is, really think of the context the agent is going to see as it's working as runtime training data. The word prompt, I think, trivializes it a bit. Right? At the end of the day, this is the data that the Agent is being trained on at runtime not literally trained on, but effectively trained on. And so you want to care about every single token and understand how that data causes the Agent to behave in certain ways. So, when you're Putting behaviors into production. It might be very difficult for one behavior to put it in exactly one, you know, paragraph that you know exactly, because, again, the behavior is all of the data at once, not just one thing. And one of the things that we developed that basis was this concept of a behavior spec. And the idea of a behavior spec is essentially saying, before we update the context, or write the context, what do we actually want? What is our intent? You can kind of think of this a little bit as, like, a product spec for the agent era. And you can't write you shouldn't be writing every detail down. You should be taking the most important opinions and putting it. Because if you put everything, you'll be duplicating a lot of states. So you define the ones that matter the most. Most. And then you can use them as a way to align your team, from a spec perspective. But then you can also use them as a rubric. And you can say, in this particular trajectory, whether that Is in any trajectory that does not have a ground truth, did the agent get into a situation in which this behavior was relevant? And then if so, did it actually exhibit the behavior? And this way, you can scale the amount of signal that you get on your trajectories far larger than trying to increase the amount of outcome based Evals, Ground Truths, and Rubrics that you have. Those are really important. But you cannot scale those anywhere near as fast as you can scale the ability to supervise what the Behaviors of the agent are. Let us just take a quick example here. This is actually the second concept, which is called forking, which we can combine here with behavior specs to think about. Forking if you have ever used Cloud Code or Codecs, the Web Apps, there is this little fork icon under each message, where you can essentially create a copy of the Agent using the same thing, which works really well on your local computer. It is hard to build a system that does this in the Out perfectly with recreating the database environment and recreating the, the prompts, all the different environments. But forking is another tool that we use to check the behavior. So a flow that you can imagine is saying, we had a trajectory could be in an eval set, it could be in testing or in production and it failed the behavior eval. And now you're trying to understand why. Why did it fail the behavior eval? Why did it not exhibit The behavior. And then you can go into the trajectory, go right before the point in which the agent made the decision that was in contrast to the behavior assuming, again, your behavior is good, and it is actually producing signal and then you can fork it. And when you fork it, you can ask the agent, or you would fork it right after it made the decision. You could ask it, why did you do that? Or you can go and update the context of the agent, and see if that improves the behavior. So with forking, you can start to actually take the signal and start to figure out, okay, how can we then improve the harness to now leverage the signal? Versus, if you're just looking at signal from an outcome based eval, you can't do that anywhere near as easily, because it failed the outcome. How do you know why? You have potentially 1,000 Inference calls in your five hour long Agent trajectory run. And then we have the last tool that I would like to talk about this is actually my favorite which is called Simulations. This is going to sound silly, but I promise it's very effective, which is that you can, when working with coding agents, ask the coding agent to make a sub agent that pretends to be your production agent. And you have to tell it to not cheat. And you give it the system prompt and the skills, and you render everything perfectly. And what you can do is, you can ask it what it would do in some scenario. You can give it the situation. And see if you can recreate the situation that you saw in your Eval or in production. And if you can recreate it, now you can tweak your context to To see if the sub agent would change its behavior based on your Tweaks. And the reason to do this is obviously far worse than forking. But you do this because this is much cheaper, much faster, and allows you, as an Agent Builder, to really easily iterate on ideas without needing to go through the expensive process of actually forking an entire environment. And you can do this today without building anything. I did this for literally a year, year and a half plus, before we built forking. Into our internal systems. And the main idea here is understanding that, if you look at the distribution from earlier, you're trying to understand, is the is the agent generalizing to that distribution? And so you don't know purely based on the outcome eval. So you can try to choose the best outcome eval that represent the curve, but you don't really know. And so supervising the process through some of these methods gives you some additional signal in order to understand if it's general. Not going to be able to develop that signal. And now, if we go back to the firm, we talked about supervision. We talked about how the agent, if he's opinionated, can be more reliable and actually can be simple. What that means is the agent maybe it starts doing something narrow. But because people will work with it, they see that it's easy to use, and they see that it's reliable. As it gets better and better, they will trust it more. It's just like a coworker. You You have a coworker day one. Imagine someone sends you a PR. You're not going to if it's you're looking at two axes whenever someone sends you a PR. It's these two axes. What's the blast radius of the PR? And what is your perception of how competent that person is? Right? If the least competent person at your company is sending you a PR that's editing a core service, you're going to look at it. If the most competent person is sending you a PR that's editing an internal document, you may not look at it as much. And so this idea is true with humans working with agents, just like it is with humans working with humans. And I think I just want to make one last point to close here, which is that one of the things that people say when they hear this is, well, wait a second. If you let the agents just do what they want, won't they find A better way? Won't they move 37 your work? And the answer is, they might. But that's not how you scale GEP and scale output. Nobody wants their agent to move 37 their tax return. They want it done reliably, at scale, in a way that they can trust, and in a way that their people know how to work with. And so, the thing I will leave you with is, for agents to diffuse, they need to be simple. For teams to trust them, they need to be reliable. The way you get to simple and reliable is you develop opinions on how the Agent behaves. And then opinions allow you to build that additional signal that helps you start to close the loop, so that you can achieve self improving Agents in production. Thank you very much. What if AI could do more than generate content and Actually simulate the physical world. To explore the next frontier of world models and what they could unlock, please welcome Cristobal Valenzuela, cofounder and co CEO of Runway. Hey, guys. Very, very excited to be here. Thank you for staying for the last talk. Welcome to my talk about the matrix and how we're living through it. So, you know, I've I've been working on Runway for almost a decade now. Started here in New York, actually, maybe around ten years ago. And, I I want to walk you through some of the learnings we've seen through, you know, building the company, building this research lab here in New York, and where do we see a lot of the work that we do in video, in pixel generation really moving forward. And so I don't Kind of exaggerating. I joke that this is... Might be a simulation or not. And and so for maybe for perspective, I'll walk you through, a piece of history. This is like watching, prehistoric ancient ruins. This is our first AI film we ever made. It was done in 2019. It's called, The Orange, because as you can tell very clearly, it's an orange that falls from the tree and walks through different parts of the world. This is done. This is a video model, an image model that we Back then, that was creating pixels in a 128 pixels by a 128 pixels that would then forcibly up sample to to this kind of resolution. I think of this as art now. This is this is like looking at a film from the nineteen hundreds, you know, it's it's beautiful. There's a lot of history behind it. But this is 2019. We started the company in 2018, 2017. And so we're trying to figure out if this had any lags. We're trying to convince ourselves that if you scale this, it will naturally improve. To the point where we can get somewhere close to this. This is state of the art video today. This is all generated. None of these characters are real. None of those locations are real. I'm sure you've you've... If you're probably here, you know this, by now. You probably have seen this online, on Twitter. The models are just incredibly good. And the most interesting thing is that when I look at stuff like this, I still believe this will look like the orange in just a couple more years. And so I want to walk you through, you know, how we're thinking not only Through what we're doing at Runway today, but also in in the next couple of years and the research that's shaping that. We are a company that's at the intersection of both both art and science. We have deep roots in artistic and creative communities, and we have some of the best researchers in the world working on diffusion models and large video models. And I like that combination because it allows us to, you know, bring the best of both worlds. We're we're a team of scientists and artists thinking about the problem of Scaling video and world models. But in a way, the the core idea of the company was already discovered by cinema, and by moving pictures a hundred and twenty, a hundred and thirty years ago, which is if you think about video, and maybe you remember this from, from high school, video is an optical illusion. Video is the illusion of movement that if you do if you stitch frames fast enough, you will create movement. You will create the illusion of movement. You create motion pictures. And so at Runway, we're doing kind of something very similar. We're taking that idea really to the next level. How do we do next frame prediction fast enough, scale enough, in in in such a way that we can hopefully do more than just pretty pretty and nice creative videos? And so Allotus Allotus is powered by, of course, scaling laws. I think we believed in scaling laws, not only applied for language models, but also for video. And this has been true. We're now really ramping up the amount of compute that we put behind the the models that you already have seen. And and The the interesting thing about, you know, seeing the model scale over time is the emerging properties that appear, the intelligence that appears when you start taking this very simple idea that cinema found itself, figuring out a hundred and twenty years ago, which is next frame prediction, and taking next frame prediction really to the next level. Right? So if I take an image like this, which is not real, those two people are not real, and I just keep on predicting what the next frame would look like based on the previous frames, and if I do it fast enough If I do it with a network that's big enough, I'll get movement, I'll get deletion of movement. I'll get something that looks so real to us that we we humans can distinguish. I I look at that video and it's... I've been doing this for ten years. I can't tell you if that's real or not. And we measure that. We've actually measured and quant... Tried to quantify, if people can distinguish between real and general videos. And I think that that Turing task we already passed, people are not able to distinguish. But the more interesting thing is here is that really what's happening as you predict next frames You're learning a lot more than just pretty frames. You're learning about physics. You're learning about object permanence. You're learning about gravity. You're learning about occlusion. You're learning about movements. You're learning about how the objects work in the real world, And you're doing that all through just video, through images, through next frame prediction. So if you look at a complex thing like this, a good video model for it to understand how to make this a consistent video will need to naturally understand. And, what all these objects are, how each of them relate to one another, how a camel might probably, stand on a boat like that, what water in a container looks like. And if I do all of that together and I do it fast enough, I'll get, hopefully, a consistent video that looks real to us. So a lot of the benchmarks that we use to evaluate our video models are not only creative, it's creative basically means do you like it or do you don't, are actually based on, on physics. On geometries, on how objects behave in the real world and how those objects should behave, and also simulator or digital worlds. So you take that idea of an extreme prediction and you do it fast enough and you get, of course, now something that looks more like what we should expect from a video application or a game application. Again, this is just just video. Right? And you take you take that idea to a next level. What happens if we start this idea of an extra prediction but do it in the context of a three d environment or something that looks similar to a game? You can start We're getting closer to a physics engine, a simulation system. But the simulation system has no physics loss. We didn't describe, what the behavior of the object should be. The model just knows how to do that, by poor force of of, of scale. And this kind of gets gets kind of interesting because you can only think about playing Worlds. This is not GTA. This is, someone playing someone else, in a environment and a software that doesn't Yes. It's all simulation of a simulation, which which gets really meta. Right? And so I'm not kidding when I said welcome to my metric simulation. But the most interesting thing for me is that everything you saw before was not real time, meaning that it took a couple minutes or seconds for those videos to be made. A lot of what we're seeing right now when we think about, you know, the next three, four years of video models and world models is to deeply think about what happens when you start not only scale In the models and making them much, much, much, bigger, which much more emerging properties, but also what happens when you start making them interactive. Right? So take the same idea of next frame prediction. You take a frame, you need to predict the next frame, then x, then x, then x. But then it can act at any point in time in that frame prediction, which means I can simulate any possible divergence or action in that world. So in this world, I could play the video, but I can also come and touch every part, any aspect of that email. And make it interactive to affect the consequences of, of that world. And it works. And of course, this is incredibly useful for a variety of different use cases, as you can imagine. We're now spinning an entire robotics lab in Runway. So if you come to our office in New York, half of it is, what looks like a Hollywood studio. Half of it is a bunch of robots walking through and moving through the office. And so we're able to make these models or start make These models, the brains of how autonomous and embodiment systems walk and reason about the world. But there's a couple of more interesting use cases that I think go beyond the two obvious one, which is video and robotics, which is what happens when you think about anything, anything, any pixel on the screen as a video model. Here's a great example. This is a video model. What you see right now, well, this is an image, first of all. That image could be real or not, but I could run one of our video models in real time. Make this an interactive application. So let's say I wanna, you know, turn the lights of the car. Maybe I wanna like turn the car around. I can just click through it. I can turn it around again, maybe open the hood and see what's inside the car. Now everything you just saw here has no physical three d representation. There's no Unity. There's no Unreal. There's no Blender. It's just a video model. A video model has been trained to understand actions and clicks in an interface. We call There's a world action model. Sorry, a world interface model. And the world interface model, well, I I feel that you guys understand now. It it's it's an interface. And it's a world model, world interface model, allows you to do very cool stuff like this. And so here's another application, another example of how you could think about using a model like this in real time. I can take an image, a representation, move objects around, and expect them to move as I was... If if as I was actually in that room. Again, Remember, there's no three d representation of this. There... This environment might not actually be have... Have been modeled before. It's just an image and the model that understands how objects move in a space because of this very simple idea that next frame prediction scale well enough can become a world model or a simulation engine. So with the eight minutes I have left, I want to show you this idea taken to another extreme. We're we're thinking not only about what happens if you can control this cool Applications and these cool interfaces. But what happens if you keep extrapolating? And I think, by the way, extrapolating might be one of the most important skills these days in in AI, in in company building. What happens over the next six months, twelve months, eighteen months may be one of the most important things to to try to predict and understand. And so we ask ourselves that question a lot at Runway. What happens... What will happen if we keep scaling the models, if we keep adding more data, if we keep making them better simulation systems, if we make them Universal simulation systems that can predict anything on a screen, literally anything on a screen. So we've been working on this idea, that we call continuum. And continuum is, the the idea of real time video models or world models taken really to the next the next degree. This is an operating system. The idea here is that you might open your computer at some point in the near future, and what you see streaming to your screen might be coming from a video model. So I'll show you next specific how this idea could could shape. This is Still very early research, but I want to show you this idea of combining video models as, as part of your operating system, as part of your daily work. So here's an example. I was researching, you know, San Francisco and the fog in San Francisco. I wanted to learn more about when and how it happens. This is structural data. This is actually a website. This is something, you can take from, from the internet. And now Contium has this idea that I can select any part of content in that website and make it a portal. We call this a portal. And the portal is a real time video. It's fully interactive and fully based on the content you're experiencing reading through. I probably won't read through all of that text. It's too much text for me. So I just wanna make a video, and I can make a video that is represented by by the text you see there. The... These portals can be three d visualizations. It can be any content that you're watching or interacting with. On the right, you can have all the visualizations and portals you've you've experienced before, anything you've been trying to learn. Basically, it's a it's a memory. It's just the disk of the imagination. Or the visualizations you've seen before. So here I was researching about batteries, and so I can go back to, the portal I had around batteries before. And since our real time interactive applications, I can click them anywhere. I can move them around. I can start asking questions, and the videos and and the content itself will change based on based on my interactions and the way I'm I'm I'm I'm managing through it. Here's another very cool example that when... The first time I saw it, when the team showed Was, like, blew my mind. Since it's a real time video, you can make it fully interactive with anything, not only with clicks, but, like, resizing the applications should also change the behavior. Right? So I can resize. I'm I'm learning about photosynthesis here. If I resize the application, I get a different experience. And since this is, again, full interactive, applications, I can click through it, navigate through parts of the cell or how photosynthesis work to get a better understanding of what I'm trying to learn and do. Again, that the one Thing that still feels to me, kind of uncertain about this is, I don't know how to call this. I don't know if this is... It doesn't feel like a visualization. It doesn't feel like a video. It doesn't feel like a game. It's something in between, and I think we need to do the collective work of figuring out how we call these kind of interactions in the first place. And I'm showing you here kind of the other thing was that you can change the structure of the prompt or what's actually creating that interaction in the first place. Here is another example. I'm gonna type something. And this is one of the ones that I feel has been the most underexplored yet. This is, we call this visual thinking. So we've connected continuum to a coding agent. And what the coding agent can do is it can use continuum to visualize your problem. So imagine you ask your coding agent to create a three d model for your bike. And instead of the model trying to just think through that via code, the model can spin a version of continuum to Think through the problem visually. So if you ever think about someone or I'm sure you have friends or maybe some... A lot of people here may be visual thinkers. What a visual thinker is is it thinks through problems and through ideas visually first, And that helps you ground and project some of the best things you want to do. And this is somehow very similar. Agents have the ability to do visual thinking. Agents can spin up a sort of imagination that you can tap and and interact with. So not only The agent can interact with it. I can also see it myself. And I can guide the agent in much better ways that it might have able to, guide it just via text alone, which is for us a very powerful idea. It's a completely new paradigm. We call this this this idea, visual visual thinking. The last example I have is how you think about taking, you know, continuum to become much more than just Using or navigating interactive structural content in in the way I've showed you before, but actually making them fully live experiences that are reactive. So in this case, I was searching for Mars. And as as Continuum understands the intention of what I'm trying to search from, it also suggests interfaces that are specifically designed in real time just for that. So here, it created for me this Mars outpost that actually should be, accurate with the data about the actual, like, robots that are In Mars right now, or with the ones that are projected maybe in 2030. And since this is fully interactive, I can interact with it. I can go to places. I can move around in the same way that I might move around, you know, in a three d system, in a three d continuum. And it works. Just fascinating. So when I say everything will be simulated, I'm really trying to understand or help not only our team, but the world. Get to the point where a lot of what we'll see today will sound obvious, hopefully, in a few more years or months. We'll look back at Continuum, and it will... Might feel very close to the video of the ORMs that we just showed you. Continuum and a lot of the things we're doing in in video and in runway, when it comes to simulation are still in very early days. But we're getting to a point where it actually it actually works and it really scales. So what we're really excited about is that for the next eight to twelve months, we're releasing most of what I showed here for everyone to use. Thank you, guys. As we come to the close of AI Engineer New York twenty twenty six, we'd like to thank everyone for being part of this incredible event. To share a few final words and officially wrap things up, please welcome back our MC, Ksenia Se from Turing Post. Whoo, clap, clap, clap. Thank you Cristobal and thank you to all of our speakers. And I would like to ask you to stay for one more minute. We have someone very special to see. We've covered a remarkable range over these three days. From building agents to changing how financial institutions work and how to simulate the world. I imagine quite a few of us have a longer list of things. To investigate than we arrived with. That seems like a, you know, reasonable outcome for a conference full of engineers. Thank you, the team who put this program together, and the crew, and our sponsors and partners and to everyone of you who came ready to share their work and ask questions and do clap, clap, clap. Please. Thank you. I hope some of the conversations that you started here Turn into things we will hear about at the next AI engineer. And I want to welcome to the stage the guy who has been behind the curation, the one and only, Suiks. She just did. Hi. Thank you so much for staying to the end and thank you Ksenia. Gabriel, Gabriel, Ksenia. Thank you guys. If you haven't seen so she I know she's very selfless and she's MCing for the second time but if you haven't seen Her videos on Turing posts, highly recommend. She has a great YouTube channel, and, she's spending a lot of time editing, that which I, really admire. Thank you so much. I hope this has been a great, few days for you. I heard a lot of feedback from sponsors, from attendees, from speakers and we're really trying to make a home in New York. This is why I'm so grateful to have Chris and Pim and all the other like great New York City AI founders. This is something that I think people don't get Enough love for in this city, which is, how much, like, real work and, frontier pioneering work gets done both in the financial institutions and just in general. AI research, AI infrastructure, and and all that good stuff. So, I won't keep you too long. I would just talk a little bit about our upcoming events. I'm wearing the shirt for code now. It's... So we're basically, you know, code is our our first home, first vertical summit. And coding agents really, I think, are, the the sort of pinnacle, right, of of Where verifiable, AI is is going. We're also confirming that we are, launching AI Engineer Europe again. That's the second year in London, in February. And, obviously, the big, big show of all... Bigger than all the other ones combined, the World's Fair. 10,000 people in San Francisco next summer. So there's a lot of AI that's gonna be happening between now and then. We may have, be all out of a job. But if you're out of a job, that means you can come to more AIEs. Right? So Anyway, we all have a good time. Thank you so much for being here. Thank you for supporting us. I hope you all made lots of friends and professional contacts. Follow the conversation back home. Right? Like, and and meet up, by... Among yourselves. That is what, it's so rewarding for us and our team to provide this community for you. So thanks for your support. See you next year.