Building an AI-Driven Kafka Cluster Migration Machine — zero downtime, zero data loss and message order intact, across ~140 services
Kafka is the backbone that carries almost every event across our platform. We moved it onto a brand-new cluster over four working days, and almost nobody noticed. No data loss, no downtime, not a single support ticket. That silence is the metric of success we look for in this kind of operation.
We relocated around 140 interdependent services across multiple environments, live and in order. We could have done it as one careful, high-stakes cutover. We chose to build something repeatable instead: a process that reads our own systems, works out the order of moves, and opens the pull requests to carry them out. It ran once per environment, and got faster each time.
Why move Kafka at all
Our largest production Kafka cluster had been left behind by the rest of our infrastructure. It was the one piece still running on the old setup after everything else had moved to the new one. It had to move too, and we modernised it on the way.
There was also a hard forcing function. Standard MSK (Managed Streaming for Apache Kafka) caps storage per broker, and we were approaching that limit. Adding capacity meant adding brokers, which raises the spend and forces a risky partition rebalance that can disturb consumption order.
MSK offered an alternative: Express brokers. They bring pay-as-you-go storage with no fixed ceiling, rebalance automatically, and replace ZooKeeper with KRaft for cluster coordination. Express tops out at Kafka 3.9 today, but 3.9 is the supported bridge to Kafka 4.0, where ZooKeeper is gone and KRaft is mandatory. That turns the eventual 4.0 upgrade into a version bump we can do in place instead of a second migration. It also cut our ongoing Kafka bill by roughly 20–25%.
Designing the strategy
Migrating between Kafka clusters without anyone noticing meant holding two hard constraints at once: zero data loss and zero downtime. A third one is easier to underestimate: preserving message order. Get order wrong and you rarely cause an outage. You corrupt customer data instead, quietly, in a way that can go unnoticed for weeks.
To guarantee that order, one rule is essential: a producer can’t move until every message it has already written has been replicated to the target. Two more sit alongside it. Services that form dependency cycles have to move as close together in time as possible, with one designated as the root to break the cycle. And because the replicator only guarantees at-least-once delivery, every consumer has to be idempotent.
That shapes the whole cutover: it runs consumer-first. Consumers are the safe side of a pipeline: with the replicator keeping the new cluster in sync, a consumer can be repointed at any time and pick up where it left off, so the risk lives entirely with producers. We migrate in reverse pipeline order and sort every service into three buckets: pure consumers (no downtime), async producer-consumers (brief downtime: stop, wait, repoint, restart), and sync producers (migrated last, in a single maintenance window). That ordering is what makes the plan hard to build: a producer can only migrate once every consumer of its topics is already on the new cluster.
| Decision | What we chose | Why |
|---|---|---|
| Replication tool | AWS MSK Replicator | For a tool we’d use once, new infrastructure wasn’t worth it. We set aside MirrorMaker 2 and Lenses K2K for that reason. |
| Cutover shape | Dual-cluster, gradual, dependency-ordered | Big-bang blue-green is almost impossible to roll back; topic-by-topic breaks pipelines mid-stream. |
| Migration order | Consumer-first | Data is already mirrored, so consumers move with zero downtime. |
| Offset strategy | latest |
earliest would re-consume the whole history: enormous, slow, expensive. |
| Target brokers | Express + KRaft | Removes the storage ceiling, rebalances automatically, cheaper to run. |
On paper the plan was clean. But it doesn’t scale by hand across 140 services and multiple environments, and that is the part we handed to AI.
How AI built and ran the migration
The buckets aren’t equally hard. Pure consumers are straightforward. Dependency cycles, where several services each produce to a topic the others consume, are uncommon but awkward: one service has to be designated as the root to break the deadlock without changing what the feature does. Sync producers mean interrupting functionality inside a maintenance window. But those are the small buckets. The weight is in the producer-consumers, about half the total, and untangling their interdependencies is the hardest part of the plan. Each one has to be placed into an ordered wave that respects the same rule, across 140 services and multiple environments, on a dependency graph that shifts every time a team ships. By hand, that is not a plan you can keep correct. AI helped in three places: understanding the system, planning the moves, and executing them.
Mapping the dependencies. The migration order is only as good as our picture of who talks to whom. We built a reporting system that reads the Kubernetes manifests straight out of our deployment repositories, finds every service that touches Kafka, and records which topics each one uses. The hard part was telling producers and consumers apart, and we solved it by cross-referencing the Kafka Admin API: a service with a topic and an active consumer-group configured is a consumer; a topic without one is a producer. The output is a machine-readable dependency graph, the single source of truth for everything downstream.
Planning the waves. From that graph, an algorithm topologically sorts the services into “waves”: each wave is a batch that can migrate in parallel, because its dependencies are handled in earlier or later waves. An earlier environment produced 10 waves; a later one only 8. Generating the plan from data paid off between those runs: the dependencies had changed in the meantime, and because we regenerated the plan from fresh reports, we picked up the change for free. The later migration went faster and the ordering guarantee still held. A hand-drawn plan would have been stale by then.
Executing the waves. Purpose-made Claude slash commands turn each wave into pull requests. They read our own documentation, open one migration PR per service, verify the rendered diff, and coordinate the responsible teams, with a companion command for the synchronous-producer maintenance window. They also keep the paperwork current as they go: live status of every PR and every phase, so anyone could see where the migration stood without asking.
The challenges
The surprise: latest and compacted topics
Our messages are serialised with a schema registry. A message carries a small schema ID, not the schema itself, and consumers use that ID to decode the payload. Those schemas live in Kafka, in a compacted topic called __schemas, whose entire value is its retained history. Replicating from latest syncs regular event topics cleanly, but it skips every message registered before the cutover, which is the wrong behaviour for a compacted topic.
SymptomReplicating from latest skips every schema registered before the cutover, so consumers on the new cluster can’t decode messages that reference an older schema ID.
TrapRe-registering through the API is no way out: the new registry can hand out different IDs, and a shifted ID decodes every referencing message incorrectly.
FixA passthrough workflow that copies each __schemas record to the target byte for byte, preserving the IDs exactly. Built for __schemas, it works for any compacted topic.
Migrating producers without breaking message order
Repointing a producer sounds like a config change and a restart, but replication comes with latency. Consider the situation shown in the following picture, where the replicator has replicated message M1 but M2 and M3 are still pending.
If we just repoint producer P and let it carry on, it writes M4 into the target while the replicator still hasn’t copied M2 and M3.
Moments later those land after M4, so the log reads M1, M4, M2, M3. We have now written a wrong order into the target topic, with almost no way to notice it at the time or to predict what it will break. The damage tends to surface long afterwards, as corrupted and possibly unrecoverable data for some customer.
Our answer was a replication barrier, wired into each rollout as an Argo PreSync hook: stop the producer, apply the new config, and resume only once the barrier passes. Because offsets aren’t comparable across clusters, the barrier snapshots the latest message timestamp per partition and waits for the target to catch up. And because shared topics create false positives, it trusts only messages stamped with MSK Replicator’s __mskmr header. It releases the rollout only once every partition has confirmed replicated messages at or beyond the snapshot.
Only then does the producer resume on the new cluster, with everything it wrote before stopping safely replicated and in order.
A migration machine
We built all of this to run more than once. The same pipeline ran across every environment, staging first so surprises turned up where they’re cheap. Because every step runs from data, nothing about it is tied to this particular migration.
We migrated the bulk of the platform during business hours. Roughly four and a half days, about two waves a day, with no service interruption.
The last wave of synchronous producers half a day over the weekend, which passed with zero incidents and small, managed partial service disruption. The migration that could have been a one-off is now something we can run again.