Moving Data Between Cloud Providers

Answer Four Questions Before Moving Anything

Copying data between providers looks like a tooling problem and is mostly a planning problem. Four answers determine the whole shape of the job.

How much is there, and in how many pieces? A terabyte in ten files behaves nothing like a terabyte in fifty million small objects. The second case is limited by request rate and listing time, not by bandwidth, and it is the one that takes far longer than anyone estimates.

Is it still changing? Static archives can be copied once and verified. Live data needs either a freeze window or a mechanism that catches up with changes after the bulk copy, which is a different tool and a different plan.

What is the acceptable downtime? Zero-downtime moves are possible and they cost complexity. A few hours of read-only mode makes the whole exercise dramatically simpler, and is often acceptable if you ask rather than assume.

What will leaving cost? Outbound traffic is billed by the source provider, and a bulk migration is the largest egress event most accounts ever generate. The pricing model is explained in the egress costs article and it deserves an estimate before the first byte moves.

Take an Inventory First

Teams routinely discover halfway through a migration that the data they planned for was two-thirds of what actually existed. Spend an afternoon counting.

The obvious items are object storage buckets, block volumes, database contents and any file shares. The ones that go missing are less obvious: container images in a registry, secrets in a vault, TLS private keys, DNS zone files, CI artefacts, log archives, queue contents, and the configuration of managed services that has no file representation at all.

For each item, record three things: its size, whether it is authoritative or reproducible, and who or what reads it. Reproducible data does not need to be transferred, only rebuilt, and that realisation often removes the largest single line from the plan.

Sizing object storage is worth doing precisely rather than by estimate. Most providers expose a storage metric that reports bucket size and object count without you listing anything, which is faster and cheaper than enumerating millions of keys. Where that is unavailable, rclone size remote:bucket will count, slowly.

Write the inventory down in a form the whole team can see. It becomes the checklist for verification later, which is the step this kind of project is most likely to skip.

Choosing the Tool for the Shape of the Data

A typical bucket copy is one command, and it is worth running with explicit concurrency rather than defaults:

rclone sync source:bucket dest:bucket --transfers=32 --checkers=64 --fast-list --progress

Where the destination is a hyperscaler, its own pull-based service is often better than pushing. Google's Storage Transfer Service and AWS DataSync both fetch from a source you nominate, which moves the scheduling, retries and parallelism into a managed service rather than onto a machine you have to babysit.

Run the transfer from a machine close to whichever side is the bottleneck, and run it inside tmux or as a system service. A multi-day copy that dies with your SSH session is a rite of passage nobody needs.

Moving Data That Will Not Sit Still

For anything actively written, the pattern is the same regardless of the technology: bulk copy first, then catch up, then switch.

For object storage, run the sync repeatedly. Each pass after the first only moves what changed, so the passes get shorter, and eventually the final pass fits inside a short maintenance window.

For relational databases the equivalent is logical replication. Take a consistent snapshot, load it at the destination, then let the replication stream apply subsequent changes until the lag is near zero. At that point you stop writes, wait for the last transactions to arrive, and repoint the application. The engine-specific mechanics are covered in the database migration strategies article.

Dual writing, where the application writes to both systems during the transition, is the other approach. It gives you an instant rollback and it introduces a real risk of the two copies diverging when one write succeeds and the other fails. Use it when rollback matters more than simplicity, and reconcile continuously rather than hoping.

Whichever you choose, decide the rollback plan before the cutover, not during it. The broader sequencing is the subject of moving between cloud providers without downtime.

Verifying That It Arrived Intact

A transfer tool reporting success means it believes it sent everything. That is a weaker claim than it sounds, particularly across providers whose APIs differ in the details.

Compare counts and totals first, because a discrepancy here is cheap to find. Then compare checksums, which is where the cross-provider awkwardness appears: the entity tag returned for a multipart upload is not a plain checksum of the contents, and it differs depending on the part size used. Tools like rclone check understand this and will fall back to downloading and hashing when the metadata cannot be trusted.

Three things deserve explicit checking beyond the bytes:

• Metadata. Content types, cache headers and custom metadata do not always survive, and a bucket of files served with the wrong content type is a subtle, visible failure.
• Access control. Permissions models differ between providers, so they are rebuilt rather than copied. Check that nothing became public in the process.
• Storage class. Objects may land in the destination's default tier rather than the equivalent of where they came from, which changes both cost and retrieval time, as the object storage explainer describes.

Keep the source intact and readable until the destination has served real traffic for long enough to trust. Deleting the old copy is the last step, not part of the cutover.

Questions About Cross-Cloud Transfers