but when you're talking to your agent and if there is sub agent that is responding to you, a sub agent can directly talk to a user. However, a function, even though it is going to give you the response, it would actually relay it back to the agent and then to the user. It wouldn't directly talk to you. The outcome on the screen looks very similar, but underneath the flow is different and it matters. It matters when you are debugging. I was so happy with my learning I decided, let's evolve this. Why not? Instead of calling the postcard agent directly, I expose it to an orchestrator agent. I expose it via agent to agent protocol, and the orchestrator agent could discover it, and call it via the orchestrator agent. So that the postcard agent now becomes a tool or like a sub agent. So a call an agent is calling an agent. My control plane has a control plane. And I did not stop there. I thought, why not expose this orchestrator agent as a API endpoint. This stopped being a demo, and it became a back end primitive. Something that is front end agnostic. Any client can call it, and that's infrastructure. And when infrastructure breaks, it doesn't look very pretty. So let's actually look at things that break and nobody show you in a demo. So I have this little postcard agent, and I thought, okay, give me the postcard for California, but send it on my work email. And it fails successfully. I'm sorry, I'm unable to send the email. The recipient is blocked. Now, that's a successful error, but is it helpful? Not really. Because it doesn't tell me if it is my Gmail settings is it my agent mails sending domain, is it a spam filter, Or there is a policy change? The error tells me nothing. And there was nothing in the logs either. It is a side effect of distributed system. This is the pattern we already know. Something upstream changes, something you cannot control, something you do not own. And the agent fails silently or cryptically. If MCP had a spec change, the same thing can happen to my tools. This is dependency drift in a distributed system. It happens in microsources, and it has happened here. So what do you need to do? You just need to version your tone contracts. Let's check out the other one. So now I know it's a postcard agent and everything, but it can still generate images because I have not got any guardrails or anything. Alright. So I ask it, can you generate a photo of a cat holding a phone? And it fails, and it errors out on maximum token limit exceeded, whatever. Token exceeded? What? Why? It took me a day to figure out that it wasn't my imagined MCP server that was failing to create the image. It was the agent's model itself which was maxing out on tokens. When it was trying to generate or render not generate, render the image. So I decided to fix it. Let's give it a URL to a storage bucket. And ask it to store there. Guess what happened? It recreated the image It definitely stored it on the bucket, but never returned me the URL. Am I supposed to view? Right? So the point being, if the image generation fails halfway, do you retry from the start? What if the email sends out, but the bucket right hasn't finished? Does the user get broken attachment? No one knows. Where does the state live? Who owns the failure? This is idempotency, distributed transactions. We have solved this years ago back in engineering, and yet here we are again solving it with a language model in the middle. Okay. Question for you guys. Let's see who is awake. Who do you think is the most underestimated problem? Anyone? No one? Okay. Authentication. Guys, authentication, that's the most underestimated problem. Because if you notice, we had four MCP servers, that means we had four auth contexts, So some could use API keys, some could use auth, agent mail can have its own token flow, and now I have a secrets management problem. A rotation problem of which credential has just expired, debugging problem. And when auth fails, the agent's error is rarely auth failed on tool x. It's usually something very very less helpful. And don't get me started on the cold starts. Every serverless tool, regardless of the cloud provider, would have this issue. Why you can fix it with some time off settings? It's more of a Band Aid fix because you can never never exactly have your latency control there. So Let's talk about the most evilest thing, hallucination. I'm glad I haven't asked any food recommendations or about edible foods at least. But hallucination in a multi tool agent isn't always this obvious. For example, in my case, when I was trying to create this postcard agent, I gave the weather MCP server URL and forget forgot the slash MCP at the end. Rookie mistake, I know. But my agent never told me that. It tried all the random tool calls only to hallucinate and tell me to try a different location. I found it I found the error buried in the logs somewhere in the terminal that showed me the light. So hallucination isn't always the model making things up. Sometimes, it's also the model confidently reasoning about the capabilities it doesn't really have right now. You can't really trust the model. And so you need observability. Log everything from day one, not as an afterthought. And if you're not testing your agents' tool calling behaviors systematically, my friends, you are flying blind. Now, there's so much that I would like to cover, and I can watch all these emerge in the documentation, in the agent development kit, and across the ecosystem. To be honest, I haven't built all of them, but but I can see their shape. And it looks like a distributed system pattern that we already know. By the way, both Google and AWS have announced managed platforms to solve exactly these problems. Memory, auth, observability, evaluation, governance. This list of uncharted territory is at being charted right now by these cloud providers. Which tells you everything about how real these problems are. Okay. Alright. So what did I really build? I didn't build three different things. I just built one thing, and kept asking what it needed to be. A toy on local host, an agent with agent to agent protocol, a service that is behind an API endpoint. And each step wasn't about the model, getting smarter, it was about the architecture getting more honest. The AI part is the easy part. Everything around it the deployment, the state, the failures, the odds, the contracts between services, that's engineering, and that's really hard. But we already know how to do it. So for all the engineers in the room, if you are building agents, you are building distributed systems. Treat them that way from day one. Version your tool contracts. Handle your failures explicit log everything. Don't wait for the production architecture to find things. And for everyone else, if someone shows you an agent demo and it looks magical and shiny, ask them what happens when tool number three fails? Ask them how they are doing the authentication. Ask them where the state lives, and the answers will tell you if they have a demo or a product My agent still forgets who I am. The memory problem, it's quite real. I tried Vertex AI engine, it's promising, but complex. I tried Bedrock, powerful, but the role assumption across the environments is generally tricky. I'm still working on it. But I now know what kind of problem it is. I started with a toy that made me feel like a wizard. I still think it's magic. I just now know what parts are engineering. AI agents aren't magic. They're distributed systems with better marketing. Thank you. I've feel wary seeing a talk in Melbourne. It's got mushrooms in it, and then we're talking hallucinations. And things, and it's all good. That's the end. That's the end of two days of AI engineer in Melbourne. This is the the bit where I have to channel my inner John Allsop to try to remember all the things that he does. First of all, thank you to all the speakers from not just our last group that are here, but all of the ones who've been here over the last couple of days because it's a really difficult thing to come up on stage and talk about something that you think you know. So round of applause, please, for everybody. Conferences like this don't come together by accident. You would see all the many red shirt people around the place and others who are not, like, the volunteers who actually make this happen. Are absolutely essential. I'm in that lucky position because I not John Allsopp that I can actually thank John because this would not exist without him. So usually, it's him ups here on stage thanking everyone else. So revenge. The final I mean, there's venue stuff. Thank you venue stuff. Yes. Definitely. Do not forget them. But for everything else that we put together to make this happen, it would not exist if not for every single one of you actually coming here. Every single ticket that sold is just the pat on the back that we're actually trying to do something interesting here. Hopefully, you're running into memory overruns after two days of many talks. But thank you everybody who's actually come to this conference. Or if you're watching online as well, it's very much appreciated, and we could not be here and would not be here without your interest. Now some housekeeping. Because Melbourne is a lovely city with lovely weather and it's always nice, we've turned on some of our most lovely weather at the moment. Fortunately, if you've got some, if you've cloak some things at the Zinc, all those things have been brought up here, so you don't have to go back down there. So well done to the people who did that. That's some good thinking. So it's bucketing down outside right now. But the last thing is we get to have another drink. Keep your badges on you and head over to Glass House, and we'll see you there and talk about all I'll try sorry. What am I reading? I'm reading John's terrible notes just after I thank you as well. Head over to the Transport Hotel, which is just that way across Fed Square. So we'll see you there. Otherwise, we'll see you at the next conference. Thank you, everyone.