AI Has No Playbook—Here's What's Emerging

Four months ago, I outlined three streams for transforming our business with AI: build, upskill, and embed. What we've learned since starts with a shift in thinking.

We spent six years wrestling startup chaos into scalable systems. AI has brought the chaos back.

Every few weeks, a new tool is released that rewrites yesterday’s rules (or at least reworks images on X). Markets swing between “AI bubble bursting” headlines, record ad revenues, and panic about disintermediation at demos. Even experts are puzzled by the gap between AI’s obvious power and its measurable economic impact.

It’s chaotic, and I love it, because this is where new things emerge. But an age-old pattern is repeating: you need balance between order and chaos to navigate it.

End of the Age of Scaling

For six years at loveholidays, we followed well-trodden paths to create order from startup chaos and help the business scale. Building a single scalable platform, against a standard set of principles, with the people and structure that let us move fast.

AI has flipped us back to a chaotic state where the playbooks that scaled us no longer apply.

Just as our ordered approach to scaling needed a little chaos to allow new ideas to emerge, this chaotic environment needs order to identify new capabilities and let them take hold.

The yin-yang in the Taijitu captures this perfectly. Each phase contains the seed of the other. The line connecting them represents the balance that prevents stagnating in pure order or fragmenting in pure chaos.

The platform we spent years shaping is foundational to finding that balance. It’s fast, flexible, focused on our business, with the observability to understand impact. Precisely the environment to navigate between order and chaos.

The vision we created for our platform also plays a vital role. We declared our intention to “Build the general intelligence for travel.” An AI that understands travel the way a great agent does, connecting customers to supply, and improving with every interaction. That vision serves as our city on the hill, a clear north star for navigating chaos.

Our platform and vision created the structure. That structure revealed what we actually needed: a new way of thinking.

The Skill AI Rewards

Systems thinking separates those who harness AI from those who treat it as a better way to generate emails or answer questions. AI delivers the biggest benefit to those who see how humans, machines, and processes fit together.

Here’s what that looks like:

We operate at scale, serving hundreds of thousands of hotels across multiple markets and languages. We’re heading toward 240 customer cohorts, each looking at the same hotel with different needs. German families scan for kids’ clubs. Whilst Dutch couples hunt adults-only retreats.

The obvious move is to use AI to write descriptions faster, but that’s the wrong way to think about it. One description can’t serve eight audiences, and creating personalised variants manually is impossible at this scale.

Instead, we thought more systemically and created orchestration with AI embedded.

The system ingests content (reviews, amenities, location data) and applies our brand guidelines. It generates highlights based on who’s searching. Families see kids’ clubs. Couples see the quiet poolside bar.

We reviewed every output at first to establish quality standards. Then we built an AI scorer that validates its own work against those standards. Self-correcting quality control.

As I heard someone say recently: “Progress is built on all the things you don’t have to do.”

Measuring Through It

To separate signal from noise, we track what matters: how AI changes behaviour, speeds delivery, and where it falls short.

We’re spearheading this approach in engineering, where seven years of platform data lets us see the impact of changes over time.

50% of our commits are now AI-assisted. But the skill isn’t prompting; it’s decomposing problems so AI and humans complement each other. We’re still writing code, but the work shifted from typing to directing: breaking down problems, guiding AI, judging outputs, refining approaches.

As AI commits climb, we track correlations with our core engineering metrics: deployment frequency, change failure rate, and MTTR. These metrics predict business performance rather than just measuring activity.

We’ve built a dedicated dashboard to monitor engineering impact, drawing on years of collected data. These foundations enable us to conduct meaningful research.

We’ve been using CodeScene (inspired by Adam Tornhill’s “Your Code as a Crime Scene”) to measure code quality for years. Their hotspot and code health metrics gave us visibility into our codebase. Integrated with all our other metrics, it allowed us to measure the impact of AI-assisted development.

In a recent interview, Stuart Caborn, our Distinguished Engineer, covered how we found that teams using AI assistance maintained code quality while dramatically increasing output.

These metrics show what actually works. Proof, not opinions. Without them, individual experiments stay isolated. With them, learning compounds across teams.

Which makes it particularly embarrassing that I spent a year confidently citing completely made-up numbers.

My Lesson

For a year, I told people we process “six quadrillion packages: the same number as all the grains of sand on every beach.”

Claude 3.5 gave me that line and I fell in love with it. Six quadrillion packages, the same as every grain of sand on every beach. Such a cool visual that I repeated it everywhere: investors, conferences, social media, and boards.

Then the upgraded Claude 4 corrected it. It was actually one beach, not all of them. To be fair, that one beach is the biggest in England (seven miles of sand at 30cm depth). But I was orders of magnitude wrong.

Whoops.

It’s great that I’m the example of how even people building AI-native capabilities get burned. The tools are powerful and fallible. This is why systems thinking matters: it’s not about trusting AI or distrusting it, but building verification into your process from the start.

The Story So Far

The three streams we implemented to navigate our response to AI are taking shape.

Systems thinking matters. Seeing how humans, machines, and processes fit together. But thinking alone isn’t enough. You need measurement to prove it works, and verification built into your process from the start.

What’s emerging is less about any single technique and more about seeing patterns. The content personalisation challenge looks different from the engineering measurement challenge. Still, the underlying structure is similar: orchestrate the system, measure the impact, verify the results.

We’re still early in understanding this. But the skill isn’t mastering individual AI tools. It’s learning to recognise how solutions transfer between contexts.

More on that soon.

For now, we’re sharing what we learn. We’d love to hear what patterns you’re seeing.