NVMe Storage on Cloud VMs: What Performance to Expect

NVMe Is a Protocol, Not a Performance Promise

Marketing pages use the word as though it were a speed rating. It is not. NVMe is the command interface between a host and a solid state device, designed for flash instead of inherited from spinning disks.

What it replaced explains why it matters. The older SATA path offered a single command queue holding a few dozen entries, which was generous for a mechanical drive and absurdly restrictive for flash. NVMe speaks directly over PCIe with many deep queues, so a device can service large numbers of requests concurrently with far less per-command overhead.

The practical effect is not mainly about sequential throughput, which SATA flash already handled respectably. It is about concurrency and latency under load: many small reads at once, which is exactly the access pattern of a database.

So when a provider labels a plan NVMe, the correct reading is that the underlying media is modern flash on a modern interface. What you actually experience depends on something else entirely, covered next.

The Distinction That Decides Everything

There are two completely different things a cloud provider might mean, and confusing them causes both disappointment and data loss.

Local NVMe is a device physically inside the host your instance runs on. The path from your process to the flash is short, so latency is low and there is no network in the way. It is also tied to that specific host.

Network attached storage is a volume assembled from a storage fleet and presented to your instance over the provider's network. It may well be built from NVMe devices, and providers say so, but every request crosses a network before it reaches them.

AWS names the local kind instance store and states plainly in the instance store documentation that data does not persist when the instance stops or the underlying hardware fails. Google Cloud calls it Local SSD and documents the same lifecycle in its Local SSD guide. Developer clouds mostly sell network attached volumes plus an NVMe-backed boot disk on the host.

The Ceiling Is Usually Your Instance, Not the Disk

A frequent and frustrating discovery: you attach a fast volume, benchmark it, and get numbers far below what the volume is supposed to deliver. The disk is rarely the constraint.

Network attached storage is governed by several limits stacked on top of each other, and the lowest one wins.

• The volume's own limit, which on most platforms scales with provisioned size, or is purchased separately on higher tiers.
• The instance's storage limit, a separate ceiling attached to the instance type. Small instances cap storage throughput well below what a large volume can serve.
• Burst credits on entry-level volume types, which deliver good performance briefly and then fall to a much lower baseline. Short benchmarks measure the burst and mislead you completely.
• Queue depth in your application. A single-threaded process issuing one request at a time cannot use a device built for concurrency, no matter how fast that device is.

Two implications follow. Size the volume for the performance tier as well as the capacity, since on many platforms they are the same dial. And check the instance type's storage figures before blaming storage, because moving to a larger instance often fixes a disk problem.

The wider question of which storage model suits which workload is covered in block storage versus object storage.

Measuring It Honestly

Numbers copied from a provider page tell you about the product. Numbers from your own instance tell you about your situation, and the gap between them is usually informative.

Use fio, because it can reproduce a realistic access pattern rather than a best case. A random read test with real concurrency looks like this:

fio --name=randread --rw=randread --bs=4k --iodepth=32 --numjobs=4 --size=4G --runtime=120 --time_based --direct=1 --group_reporting

Four things make the result trustworthy. Use --direct=1 so the page cache is bypassed and you measure the device rather than memory. Choose a working set larger than the instance's RAM. Run for long enough to exhaust any burst allowance. And read the latency percentiles, not the average, because the tail is what users feel.

While a real workload runs, iostat -x 1 is the better tool. The %util, average queue size and await columns together tell you whether the device is saturated or merely busy, which a throughput figure alone never does.

Do not publish or plan around a single run. Storage performance varies by region, by instance type and by time of day, and the honest conclusion is usually a range rather than a number.

Getting the Most From What You Have

A handful of choices at setup time matter more than any later tuning.

Pick the filesystem deliberately and align it with the workload. ext4 is the safe default; XFS handles large files and high parallelism well. Both are fine on NVMe, and the difference between them is smaller than the difference made by the next two items.

Mount with noatime so reads stop generating metadata writes. On a read-heavy service this is close to free performance.

Check the I/O scheduler. For NVMe devices, none is usually correct, because the device reorders internally and the kernel scheduler only adds latency. Read the current value with cat /sys/block/nvme0n1/queue/scheduler.

Then look at the database, which is where most of the benefit lands. Increase the number of concurrent I/O requests the engine will issue, since defaults were often chosen for mechanical disks. Keep write-ahead logs and data files on storage that is actually fast rather than on the boot volume by accident. The PostgreSQL guide covers the specific settings.

Finally, leave headroom. Flash devices slow down as they fill, because garbage collection has less free space to work with. Running a volume close to capacity is a performance decision as well as a risk.

Choosing a Plan Without Being Sold To

Provider pages describe storage in whichever terms flatter the product, so it is worth normalising the claims to a short list of questions.

Is the storage local or network attached, and does it survive a stop? Does performance scale with volume size, or is it purchased separately? Is there a burst mechanism, and what is the baseline once it runs out? What is the instance type's own storage ceiling? And is the fast disk the boot volume or an additional one?

The answers vary more among the developer clouds than their marketing suggests. UpCloud built its reputation on a storage backend it sells as MaxIOPS, and Vultr separates its plan families partly by the storage and CPU pairing, which the Vultr plans article unpacks. An UpCloud account and a Vultr account are both reasonable places to run your own fio comparison, and the DigitalOcean and UpCloud comparison covers how storage performance figures in the wider choice.

Whatever you choose, remember that fast local storage raises the stakes on backups rather than lowering them, which is the subject of the backup strategies guide.

Key Takeaways

• NVMe describes the interface. It predicts concurrency and latency behaviour, not a specific speed.
• Local NVMe is fast and ephemeral; network attached volumes persist, snapshot and reattach. Know which you bought.
• The binding limit is often the instance type or a burst allowance, not the volume.
• Benchmark with fio using direct I/O, a working set larger than RAM, and a run long enough to exhaust bursting.
• Mount with noatime, set the scheduler to none on NVMe, and raise the database's I/O concurrency from its mechanical-disk defaults.
• Keep free space. A nearly full flash device is a slower flash device.