"When you learn to fly, autopilot is the last thing they teach you." That's a phrase I uttered at a Cloudorizon all-hands, when we were sharing war stories of customers who had got themselves into trouble using AI assistance when building integrations. It stopped the room and made everyone think. Was it even true? Does comparing our rush into AI-assisted development with one of the safest industries on the planet tell us anything? Matthew Syed argues in his 2015 book, Black Box Thinking, that everyone has something to learn from the airline industry. In answer to my first question, I asked my friend Max, who recently qualified for his pilot's licence. His answer: autopilot is "not permitted to be used properly before obtaining a licence".
Many of our customers use Workato to build and run integrations, automations, AI agents and MCP servers. Back in the summer, Workato started rolling out AIRO, a set of agents that take over the planning and building for you. And I have to say, it’s great. I use it wherever it’s enabled, to handle tedious refactoring at scale or to quickly understand implementations I’m seeing for the first time. That said, within a week of its release, my team had two usage stories that raised eyebrows.
The first was at a client we provide extended onboarding oversight for. One of their engineers had a very normal question for me: “I’ve made this recipe. How do I make it work in batches to cut down on cost?” But when I looked at what they had implemented, it was already working in batches. The engineer hadn’t really built the recipe: AIRO had, and they didn’t understand what it had done. No harm, no foul: the pilot had their instructor sitting in the seat next to them to correct their course. If I hadn’t been there, it could have ended up as a loop over the batches, resulting in thousands of extra rows written and mass duplication in a downstream system.
Mode confusion is a term in the aviation industry for when a pilot thinks the autopilot is doing one thing when it is actually set to do something completely different. That’s exactly what happened to the client’s engineer: they thought that row-by-row logic had been built when, in fact, batching had already been implemented. You can roll back a recipe. You can’t roll back ten thousand duplicate rows in a downstream system, or a decision an agent has already taken on someone’s behalf.
Both the FAA and EASA warned in 2013 that pilots were leaning too heavily on automation and losing their hand-flying skills. That doesn’t mean automation is bad: regulators agree it has improved safety, but it’s taught deliberately to people who can already fly. Max points to a time when his altimeter was wrong because he had entered the wrong air pressure for that day. He spotted it because he knew what the reading should be. Autopilot only works when everyone, from pilot to air traffic control, knows what they are doing, so small mistakes are caught quickly before they become disasters. Max is confident he would spot a faulty autopilot: “I could easily identify using instruments for height, direction, yaw and roll if the autopilot was not working correctly.” We should be able to say the same for those of us using AI to build mission-critical products.
In the second story, what AIRO built was not wrong, and that was the problem. My engineer, while investigating a bug in a new integration, stumbled across a Workato custom connector that had been built by someone who didn’t normally build connectors. They’d simply kept prompting AIRO until something worked. It had been tested, gone through change control and been pushed into production, but what was under the hood was messy. Change control checked the connector worked, not that it was built well. Schemas too tightly formatted to read, duplicated logic and overlapping functions meant that, one day, someone was going to have a hard time maintaining it. Perhaps worse than all of that, the connector just wrapped the API, preserving all of the complication the connector layer is supposed to alleviate.
In defence of AIRO, my engineer thought that if they’d been more experienced with Workato, or programming in general, the person in question could have prompted AIRO to fix the issues. That’s precisely the point. Workato’s own documentation points you to testing everything AIRO builds, but how would they do that without knowing what correct, or good, looks like? The connector described above is still running in production. Nobody at the organisation has time to go back and fix it, so it sits there, a bill waiting to come due. Technical debt if ever I’ve seen it.
Going back to the world of aviation: it’s full of checklists, and they’re drummed into pilots from day one. As Max puts it: “If I lose my place on my list, I have been taught to start again from the top.” In Black Box Thinking, Syed tells the story of Dr Peter Pronovost, who introduced a simple five-point checklist for a routine hospital procedure and, crucially, gave nurses the authority to stop doctors who skipped a step. The infection rate fell from 11 per cent to zero. Rolled out across Michigan, it saved an estimated 1,500 lives and $100 million in eighteen months. The checklist mattered, but so did the authority to use it.
As an IT leader, you should have your own frameworks and guardrails in place to make your AI building safe. A simple checklist is a good place to start:
The checklist covers the thing that was built. It doesn’t cover the person who built it. Who decided that the engineer who used AI to build a product or feature was qualified to build it? In most teams, nobody did. A flying instructor decides when a pilot is ready for their first solo flight, whereas building with an agent tends to be a solo affair from day one. Someone in your organisation should own that decision: who is cleared to build what, who reviews AI-assisted work, and who has the authority to stop it. That is an operating model question, not a tooling one. And if your juniors only ever prompt, who in three years’ time will know what the instruments should be reading?
To be clear, this is not a rant against AIRO or any other way of producing work with AI. I use Claude Code for an ever-growing portion of what I produce. I’m also not saying that trainee pilots never use autopilot; it’s a great tool when they’re stretched, as long as they are still the pilot. The same goes for building with AI agents: they’re a great tool, so long as someone remains the pilot. AI makes a good builder great. It doesn’t make a passenger a pilot.
And the stakes are rising. AIRO doesn’t only build recipes and connectors; it builds agents, and an agent decides at runtime what to call and on whose behalf. A badly built recipe fails in ways you can predict. An agent with the wrong access fails in ways nobody planned for.
So autopilot is great when the pilot is well trained, has well-thought-out checklists to follow and plans their route around controlled airspace. They know where Heathrow is; they’re not going to let autopilot fly them in a straight line through its airspace. But where does that leave us with our newfound AI building capabilities? Who’s going to check we’re not flying straight into Heathrow? The answers have been in IT all along. Your checklists are your change control processes and your governance, your strategy is your bearing, and your reference architecture is what stops you flying into the flight path of a 747. Getting your enterprise architecture right is the difference between safety and danger. It’s also the difference between an over-governed, unusable bureaucracy and a team that can genuinely move faster.
Before you move to AI-assisted development, decide the things it can’t decide for you. Who is allowed to build what? Who reviews the work, and against which standards? What may an agent call, and as whom? Who owns all of it after go-live? That isn’t bureaucracy; it’s the flight plan.
So who’s flying your integrations, and has anyone checked the route? If you’re at World of Workato in London next week and AI is on your roadmap, come and find me. I’m happy to talk through the playbooks, standards and operating model that let your team have the speed without the bill.