# CTO, Blackstone — John Stecher, Scott Worland — session 2026-10-14T18:05:00.000Z → 2026-10-14T18:23:00.000Z

_51 transcript lines · 0 slides · source: full recording_

## Transcript

**[00:00:00 · 0]** And, you know, as a big asset manager, we have a ton of data, proprietary data, public data all woven together. We have to move that into... Out of systems into something that we can really leverage from an AI perspective. I think the second thing that, you know, has nothing to do with AI, but you can leverage AI to actually figure out how to do the best possible way, is straight through processing still matters. I'm always shocked at how everybody just talks about, like, agents as being the biggest differentiator in the world.

**[00:00:26 · 0]** The thing that I've leveraged myself using factory and a couple other ways to go off and build code inside of the firm is, like, taking time, having agents help to craft something for me, understand the workflows out there that people are doing, and then make sure that I'm, like, actually writing code to optimize for straight through processing, and then using agents to inject in intelligence where necessary, where humans were making decisions in the past for a variety of rote work things. So that's really, like, the two key elements that I spent a ton of time thinking about every single day. And then the biggest thing from an agent perspective is really stepping back and looking at, like, what is work that we want to build agents to do better, to offload, like, intellectual cycles from our investment professionals, our finance professionals, our engineers, such that they can do higher value work. And there's a ton of stuff when you think about the software development life cycle, writing test cases, functional test cases, performance test cases, versioning forward libraries underneath applications. It's the bane of every engineer's existence.

**[00:01:24 · 0]** I've hated it for twenty seven years that I've been doing this. Having an agent be able to go off and do these things for me and renovate my code and my source code has made all the difference in the world for the velocity that we deliver at. So how are you thinking about it at Norm?

**[00:01:36 · 1]** Yeah. So norm was founded in 2023. So, you know, that's three or, I guess, four years after GPT-one, but still a year before there was, like, chain of thought. So it was, like, an interesting time in terms of development. I remember when we had our first team get together, we were at Stanford's campus for a week.

**[00:01:53 · 1]** I mean, I'm probably misremembering the exact timeline, but it felt like we went from four ks to eight ks to 128 ks context windows, and then we had multimodal. And everything just felt like it was kind of coming all at once. And then there was a bit of a time where things did slow down when it was coming from the frontier. And so I think, like a lot of companies in early twenty twenty four, we were trying to deploy these models as part of our workflow, and as part of our production software. And so we ended up adding a lot of structure to the system.

**[00:02:20 · 1]** This comes from all different parts of the stack. So whether that was from how we went and took the content that we knew needed to be, like, eventually added the context to the models, and we had very specific ways in which we did that. Or if there was, like, you know, just the sequencing of the models, if there was steps we needed to do in terms of looking for consistency, and then ultimately distilling everything down into some useful output, we ended up with something that looked like a large MapReduce problem, effectively, right? Whereas we had a lot of scaffolding for the agents, and we were very particular about how these well, I would say it's agents, really, just how these models ran. It was much more like a workflow.

**[00:02:55 · 1]** And what this allowed us to do was get really high quality, and also with accurate, very accurate, but also just very consistent, which is very important for us in the regulatory space outputs. But then one fortuitous thing as part of doing this, because we wanted to have some way to regression test this big system, we ended up with a really large set of evaluation samples. We had tens of thousands, actually, of, like, hand labeled examples by experts. And so as soon as we moved to a place where it was, like, clear that agents were actually a thing, and by agents, I simply mean, you know, given, like, an LOM, a system, or given it a set of tools that it can use, give it some goal, and give it some way to verify itself against that goal, allowing it just to go figure out how to do the thing in the best way possible. We were able to just take this benchmark set and just immediately run on top of it, with some of these more agent take approaches.

**[00:03:42 · 1]** And we saw, you know, I think a lot of people had this moment where you're staring at hundreds of thousands of lines of code your team had really thoughtfully written, and realizing most of it was pretty obsolete, and was just kind of actually in the way. And so to update quickly on this, what we did was, we took I got the engineering leadership team. We actually rented a Sprinter van, and we drove to Upstate New York. It was actually after a big snowstorm. We had an Airbnb, and it was like, forget Norm for a minute.

**[00:04:06 · 1]** Let's just say, what would we build today as far as a platform, knowing what we know right now, if we were gonna be starting over? And then from that, we left with basically a set of specs about what this platform needed to be able to do to enable two things. One, deliver Adjinta experiences to our clients, and then also, it needed to be amenable to having AI as co developers alongside of our own development team. And that's what we've been executing against ever since then. So I think this do you have a question?

**[00:04:33 · 1]** No, I was

**[00:04:34 · 0]** going to say, I think you hit on the key points, Scott, where it says the eval side of the fence that you were talking about before. If you're working in financial services, like, the quality bar for what agents have to deliver on is very, very high. And so I think really thinking through, like, what is your eval set for how you basically test your agents, what they're actually whatever work they're performing, that the output makes sense and it's consistent and repeatable. And I've seen across the board, as model versions change underneath the covers, you get different answers. You get skew in terms of the accuracy of information that comes back.

**[00:05:07 · 0]** So having good eval so that you can run those consistently and really measure the health of a system is paramount to success. Same thing happens if harnesses change that you're running stuff on top of. You'll get different answers. Data changes, different answers. So it's something where I think if you're traffic in financial services, like, the duty to be accurate is, like, up here.

**[00:05:26 · 0]** And hitting that is, like, something, like, we think about all the time.

**[00:05:29 · 1]** Definitely. I think this leads nicely into this concept that we've been talking about, just, like, with the bitter lesson. So, like, everyone's familiar with it. And, you know, there's this idea that these handcrafted, systems built by, you know, clever people thinking hard about the problem will oftentimes eventually be dominated by systems that scale with compute. Right?

**[00:05:47 · 1]** And that this is actually... I think Rich Sutton originally talked about this in 2019, but this has been since the beginning of neural networks. If you go back to, the '50s and '60s, like with Rosenblatt's little Perceptron, I think the New York Times ran an article, or somebody did, saying it was the first machine that could ever have its own original thoughts. There is actually quite a fascinating history here of how the expert systems and the connectionists and what ended up happening. A lot of it actually was highly litigated in in England, and a lot of AI research was dismantled.

**[00:06:15 · 1]** But one funny thing, if you go back to the eighties even, there was a, lead at IBM's translation lab, and he made a comment, which is very much the bitter lesson. Every time I fire a linguist, my models improve. Right? No. The point of having these, like, linguists in there and trying to think about all the rules of language versus, like, training a statistical model based on just probabilities.

**[00:06:31 · 1]** And so we speed up to today. And, like, we see this all the time. We've been touching on lots of points about this. And I'm curious. When you think about your roadmaps, when people on your team are bringing you project ideas, how do you keep this in mind as you're thinking about this for Blackstone?

**[00:06:44 · 0]** Yeah. I mean, a lot of what we do is, like, the simple equation is, is giving enough compute, every problem is solvable by hardware and software at some point in time in the future. It's... Elon talks about this energy wise, etcetera. So it's like, it's a big map.

**[00:06:58 · 0]** It's a whole bunch of stuff. How do you bring it back to reality? Like, what I spend a lot of my time doing from my seat working with engineers on my team, with product people, with our SRE function, with stakeholders, is really trying to define what is the ROI of what I'm being asked to work on. And so when I think of building agents, I think about getting data into the right spot, think about building dashboards, trading platforms, whatever it is. It's like, if I look at, like, what's the ROI that's going to be created from the actual ask that's coming in to me from a team perspective, and then, like, what's the true cost of actually building something today?

**[00:07:33 · 0]** Like, how can I balance that equation? And then I make a decision of, like, is this something where the models are going to get good enough? I'm going to have more than enough compute in six months that it's not worth trying to tackle the problem today. It's better to delay it and actually address it with one of the new models. We have made that decision before.

**[00:07:51 · 0]** It's like, let's actually wait for the model to become slightly more sophisticated so that we can solve these problems with it because it will cost less at that time. And the need for that application to be built today isn't pressing. There's no, like, real business value increase that will be created because it's produced today versus six months from now. So that's a trade off that we actually make regularly and I think about regularly. The other thing that we do is, like, you don't want to wait too long to basically build something.

**[00:08:18 · 0]** And so you can always be waiting for the new model, waiting for the new technology, or whatever else. Like, the beautiful thing with, like, all of the various different software development harnesses today is if you lean into them, it's amazing how quick you can go from concept, mapping out your planning and concept with, like, Fable for just, hey, work with me and let's define this with the product person or with the stakeholder, to actually writing code, to having a workable proof of concept. And when I say workable proof of concept, it's something people could click through, interact with, whatever else it is. Doesn't have cybersecurity review. Doesn't have all the other things that you need to do in our industry to make sure you, like, put something into production that scales.

**[00:08:54 · 0]** But you can get to a POC or a POV very quickly. And so, like, that's one big thing I try to encourage all my engineers to do frequently is, like, just get to POC. Don't worry about PowerPoints. Don't worry about anything else. Leverage tokens, which I think it's a fallacy to be counting tokens in some way, shape, or form.

**[00:09:10 · 0]** It's better to think about producing value. Go off produce value, produce POVs, put it in front of people, iterate on it quickly, and throw it away if if you don't need it. You both spend far less time and money doing those types of things. And so that's, like, a big way that we think about trying to solve it. It doesn't maps into the parallel, like, the, you know, the how do you deal with increasing compute, making it easier to do these things, but But I think in a much shorter time frame.

**[00:09:34 · 1]** Yeah. It's so interesting, because it's like, you know, I mean, you touched on so many things that I think everybody's probably been thinking about too. And if you go back to 2023 when we were originally, like, sitting around as a small group talking about what we're building, we we were aware of the better lesson thing. We talked about it. We said, how do we not get in this trap?

**[00:09:49 · 1]** And I think at this point, it's just really evident to me that it's definitely a gradient. Right? It's not this binary thing about you can somehow build the perfect technology to index into all the up know, everything coming in from the frontier in a way that you don't have Raft to throw things away. So really, it gets down to, like, where do we want to invest heavily and where are we willing to invest conservatively because, like you said, we need something today. But at the same time, we're just aware that, like, some of these investments we make, particularly around things like we've built, you know, fairly complicated systems around all of our agent primitives, so the harness, the sandbox, how egress works, you know, all of these things.

**[00:10:22 · 1]** And we we also want to make sure that as, like, things are coming out either from, like, a Frontier Lab or from, like, a a third party, that we're able to index into that. And so I think a lot of it is just kind of when do you update. And that's one of the hardest things to do. And for me, a good example was I really updated really hard on Cloud Code, not so much that it was suddenly tools that, like, in my profession that we could use that were really incredible. What it really was for me was just this idea, because I was a bit skeptical up to that point of how we would get something that was, like, truly generalizable.

**[00:10:51 · 1]** It's like, well, what does Cloud Code give you? Well, you can write code, and then it can execute the code, and, like, basically run a computer in some verifiable way. At that moment, I was like, well, we have all the ingredients for, like, a truly generalizable intelligence, because if you can go do this at scale, then, I mean, you know, of course, like, you can basically do anything in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today. And You fast forward to now, and all these tools that people are using, these AI assistants and things, this is why they're so amazing. It's because you can ask a question it's never seen before, fully out of distribution.

**[00:11:19 · 1]** It can spin up a computer and go do this thing in the background. And for consumer, it's amazing. And we're also seeing this now within the enterprise, too. And so for me, that moment was like, well, where are we going to invest heavily? Of course, we still need to invest in what we're building on the agent side.

**[00:11:32 · 1]** But going back to what John mentioned, how we do our data capture and distillation, and how we make this available is like state to these models, and then also just how we do our benchmarks and evaluation. With the law firm, for example, we have this really unique situation where we started this law firm from nothing. We've grown together. They've grown the norm together. So we work directly with lawyers to stand up these benchmarks and evaluations.

**[00:11:57 · 1]** And effectively, rather than think of a traditional roadmap, like when are you going to build an agent that can go do these three things, it's let's have a scorecard that determines when an agent, or when a human even, is has success... Is successful, at completing these tasks across various dimensions. And then we wanna be able to see this just improve over time. So as we make general improvements to the system, we should see an uplift to these, you know, these things like an attorney would expect to improve their workflow, but at the same time, not not getting... Kinda going back to the state that we were originally where we had added a lot of structure to it in a way that ultimately, we we found that we were actually...

**[00:12:29 · 1]** We could get even better performance if we were to remove that upfront structure. Yeah.

**[00:12:33 · 0]** I think, like, another thing that's important to think through is, like, you can produce, like I was saying, a lot more proof of concepts, proof of value very quickly. I still think, like, the hard yards for a lot of development out there is getting it to production. And what I mean by that is, like, infrastructure build out that actually runs inside of the infrastructure that's available inside of any financial services or enterprise company, for that matter out there. I think that is a challenge that a lot of folks, like, underestimate. If you're selling AI tools, that is hard on the other side, just so you know.

**[00:13:03 · 0]** I deal with it quite a bit. The other thing is is just, like, dealing with, like, the cybersecurity side of the fence and the supportability side of the fence. And so, like, one thing I've noticed, you know, the agents are getting better at this now, but it's making sure that the actual software that's being written is truly traceable and it feeds into your SRE platform so that it can be monitorable in production. Those are things still that, like, like, that people think about it. It's it's like they haven't started to scale yet with agents.

**[00:13:29 · 0]** They will. But it's something that, from my perspective, I I think about regularly.

**[00:13:34 · 1]** Makes sense. And our approach to this I'm sure a lot of other companies, and maybe even you guys, have taken this approach as well is there's parts of our code base that we still require, like, complete human oversight from when it comes to designing, building, code ownership, even improving any kind of changes to the code base. And these are like our core primitives. And these are things where, you know, basically, you can build things on top of these. But the things that you build on top of these are not allowed to, like, change these systems without having, like, human review.

**[00:14:00 · 1]** And it's because of the things you mentioned. It's like, Data security for us is as paramount as delivering quick. It's actually more important. So it depends on where you are, whether it's a consumer product, or for us, obviously working with highly sensitive data. We have to build in so much infrastructure around this and ensure that we're not ever getting out ahead of these parts of the stack, which it can be hard because maybe somebody brings to you and I'm sure you've seen this a really cool demo.

**[00:14:26 · 1]** And it's really exciting. And you're looking at it terrified. So you're like, this is very cool. But to get to this point in terms of capabilities, it would expose so much that we're not able to do yet. So I think it's just always that trade off.

**[00:14:36 · 1]** So we just, you know, basically we invest heavily in these things and then invest conservatively in places where we realize as the frontier continues to improve, we wanna make sure we have the conduits. And the way I described it actually this morning with the team is like, basically, throughout our platform, we wanna have, like, conduits for what's coming from the frontier. You don't wanna have dams where, like, the intelligence is coming in, and it can't quite get into your stack because you built things in such a way. Again, it's a great end. We're gonna get some of it wrong, some of it right, just being very cognizant of it, talking about it a lot, and making sure that our core pieces are are locked down, but then also, you know, being very open about, just rip and replace.

**[00:15:10 · 1]** I think that's, like, a thing that I think everybody's gotten really good at.

**[00:15:13 · 0]** Yeah. I I think that's, like, one of the biggest... I mean, like, I've been doing this for twenty seven years. Sometimes I feel like it's been, like, a hundred years. Other times I feel like it's been five.

**[00:15:21 · 0]** I think that the thing that's, like, really made a material difference, like, when I I look at the ability to write software today versus what I was doing ten years ago, was ten years ago, it took so much time, effort, and money in people's, like, intellectual capital to produce a piece of software. It was almost like the ship had sailed. Like, by the time you designed it and started writing code, it was out to sea, and you were gonna ride that baby for, like, as long as you could until it got to the other side of the ocean. And, like, what I've become very comfortable with today is, like, for the most part, a lot of, like, internal applications can be just thrown away. It's build them, have them work for a purpose and a point in time, but understand that they're going to disappear at some point in time now much more quickly than ever before.

**[00:16:05 · 0]** And then just think about, like, what comes next. And so I think one of the challenges with, like, sitting in my seat or any, like, good engineer seat is to be constantly questioning, like, you know, should I rewrite this? And does it cost me less to rewrite it than it costs me to keep versioning the current version forward? And that is a question that a decade ago never would have came up in anyone's head. If you went to say, I want to rewrite an internal application, your CFO would have looked at you like you were insane.

**[00:16:29 · 0]** And I think now it's a completely relevant thing that all of us should be doing because net new applications might produce way more business value at the end of the day. So it's just something I encourage folks to do. It sounds like you guys are doing quite a bit at norm.

**[00:16:41 · 1]** Yeah, exactly. And I think it's just that concept of what used to be expensive that's now cheap. And if you don't update quickly on that, then it's, you know, it... Like, you... It's incredible.

**[00:16:49 · 1]** You said it feels like 27 or 5 years or a hundred years or five years. It's like the... If you over if you over index too hard on something that was in June and now it's October, like, you know, it it feels like, in some ways, there's been, like, ten years of technological advancement. So, well, great. We only have a minute or so left.

**[00:17:04 · 1]** Is there any closing remarks?

**[00:17:05 · 0]** I mean, I think, like, the the one thing I'd, like, leave the audience on if if you want, like, you know, like, good takeaways from this, it's like, we talked about a lot of, like, very interesting things. I think, like, what makes me, like, excited to go to work is, like, it's cool to get back to engineering. One of the things that, like, I value is, like, you know, I was a hardcore low level engineer for ten years at IBM and, like, wrote, like, you know, this is way back in the day of using Notepad and VI to write, like, instruction level code for risk machines. It was, like, awesome. It's nerdy as all heck.

**[00:17:34 · 0]** No one does this stuff anymore, but it was, like, fun for me. I still enjoy those days. As you get more senior in your career, it, like, the barrier to actually writing code on a daily basis disappears, or it becomes more, like, hard to get through, like, dealing with a stream, like, processing stuff off a stream. It's just all these things are like, oh my god, I can't even get started because the scaffolding is so hard. With, like, AI, it's been, like, reinvigorating to get back and actually build stuff

**[00:17:59 · 1]** because it's

**[00:17:59 · 0]** like

## Slides