Migration Testing and Rollback Plans

Plan for the migration that goes wrong

Most migration plans describe the happy path in detail and the failure path in a sentence. That imbalance is why teams end up making consequential decisions at three in the morning, tired, with an audience.

A rollback plan is not an admission that the work is shaky. It is the thing that lets you attempt the cutover at all, because a reversible decision can be made quickly and an irreversible one cannot.

The two halves of this article belong together. Testing is what tells you whether to proceed; the rollback plan is what you do when the testing was optimistic. Neither is useful alone.

The tests that earn their place

The last row is the one most often skipped, and it is the one that matters most. A rollback path that has never been executed is a hypothesis with a reassuring name.

Proving the data is right

Application testing tends to be thorough because it is visible. Data verification is quieter and more consequential, since a functional bug is noticed and a silently wrong row is not.

Reconcile at three levels. Structural first: table counts, row counts per table, and the presence of indexes, constraints, sequences and stored routines. Then statistical: sums of numeric columns, minimum and maximum timestamps, distinct counts on key fields. Then sampled: pull a set of records from both sides and compare them field by field, including the ones with unusual characters or extreme values.

Pay attention to representation rather than just content. Time zone handling, character encoding, floating point precision and null-versus-empty-string differences all survive a row count and change the meaning of the data.

Automate these checks as a script rather than a checklist, because you will run them several times: after the rehearsal load, after the final sync, and again a day later. The strategies that produce the data in the first place are covered in database migration strategies.

The dress rehearsal

A rehearsal is the full cutover performed against a copy, with the same people, the same runbook and the same clock. Its purpose is not to prove the plan works. It is to find out where the plan is wrong while that discovery is free.

Run it end to end without skipping the boring steps, because the boring steps are where the surprises live: a credential that was only in someone's shell history, a firewall rule nobody documented, a certificate that takes longer to issue than expected.

Time every step and record the actual durations next to the estimates. Most first rehearsals overrun substantially, and the gap tells you whether the real window is realistic or wishful.

Capture the defects found and fix them before the second attempt. If the rehearsal produced a long list, run another one. Teams that skip the second rehearsal because the first one was messy have the causality backwards.

Deciding to proceed

Write the go conditions before the window opens, and phrase them as facts that can be checked rather than judgements to be argued about.

• Replication lag is zero and has been stable for an agreed interval.
• All reconciliation checks pass, with any exceptions explicitly accepted in advance by a named person.
• Backups of both environments are complete and verified within the last few hours.
• The rollback path has been rehearsed and its owner is present.
• Key people are available, including someone with authority to abandon the attempt.
• No other significant change is in flight anywhere in the estate.

Write the abort conditions with equal care, and give them times. "If we have not promoted the database by the agreed hour, we roll back" is a decision made calmly in advance. The same judgement made live, with sunk cost pressing, reliably goes the other way.

Four classes of rollback

Rollback is not one mechanism. It is four, and they become progressively harder as the cutover proceeds.

Traffic reversal is the easiest. If you shifted traffic by weighted DNS or through a proxy, you shift it back. The constraint is that DNS reversal takes effect on the resolvers' schedule, which is why a proxy layer is worth the extra hop during a migration.

Application reversal means pointing the old application stack back at the old data, which works cleanly only while the old database is still authoritative. Keep the old environment deployed and idle rather than deleted.

Reverse replication is what preserves reversibility after writes have landed on the new primary. Configure replication from new back to old immediately after promotion, so the old database stays current and remains a viable target. This step is frequently planned and rarely implemented, and it is the difference between a reversible cutover and a one-way door.

Restore from backup is the last resort and involves data loss between the backup and the failure. Know the restore duration by measurement, not estimate, because that number determines whether it is a plan or a disaster.

Writing the runbook people can follow

A cutover runbook is an operational document, not a design record. Every line should be a step with an owner, an expected duration, a verification and a note on whether it is reversible.

Keep it somewhere reachable when your systems are not: a document platform outside the estate being migrated, plus a printed or offline copy for the coordinator. A runbook stored on the infrastructure you are changing is a familiar and avoidable mistake.

Mark the point of no return explicitly, usually the first write accepted by the new primary. Everyone in the room should know when that moment passes and what changes about the available options.

Afterwards, keep watching. Monitor error rates, latency, queue depths and the invoice for a fortnight, and schedule a review of the weekly and monthly jobs that have not yet run. The wider sequencing this fits into is described in cloud migration planning, the traffic mechanics in moving between cloud providers, and the release pattern that inspires much of this thinking in blue-green deployments.

Key takeaways

• Testing decides whether to proceed; the rollback plan decides whether you can afford to try. Build both.
• Reconcile data structurally, statistically and by sampled comparison, and script the checks because you will run them repeatedly.
• Rehearse the full cutover on a copy, time every step, fix what it exposes, and rehearse the rollback too.
• Write go and abort conditions as checkable facts with times attached, before the window opens.
• Know which of the four rollback classes applies at each stage, and configure reverse replication if reversibility matters after promotion.
• Keep the runbook outside the estate being migrated, mark the point of no return, and keep watching for a fortnight afterwards.

For platform tooling, see the Route 53 failover documentation and the Azure Site Recovery overview. If the destination is still open, the AWS and Azure comparison and the Azure account listings cover what each side provides.