GPU Cloud Instances Explained

A rented accelerator with a virtual machine wrapped around it

A GPU cloud instance is an ordinary virtual machine that has one or more graphics processors passed through to it. You still get vCPUs, system memory, a boot disk and a network interface. The difference is a device on the PCI bus that your code can address through CUDA, ROCm or a higher level framework.

That framing matters because most surprises with accelerated compute are not about the accelerator. They are about the ordinary parts: the disk that is too small for a dataset, the vCPU count that starves the data loader, or the region that has none of the hardware you asked for.

Providers group these machines into named families so you can reason about them without memorizing every chip. Learning the grouping logic for two or three platforms transfers cleanly to the rest.

How the major platforms group accelerated machines

Every large provider splits accelerated capacity into at least two tiers: heavyweight multi-GPU machines aimed at model training, and lighter single-GPU or fractional machines aimed at serving, transcoding and visualization.

The custom silicon column is worth a second look. AWS designs its own training and inference chips, and they are programmed through a dedicated SDK rather than CUDA, so code portability is a real design decision rather than a footnote. Google Cloud does the same with TPUs, which are requested as their own resource type instead of as a GPU attached to a VM.

Developer clouds compress the whole picture into a short plan list. That simplicity is genuine, and it is one of the reasons teams that only need a couple of cards often prefer them, as the comparison of hyperscalers and developer clouds sets out.

Memory on the card decides what you can run at all

Ask about GPU memory before anything else. A workload either fits in the card's memory or it does not, and no amount of extra vCPU or system RAM changes that answer.

For deep learning, three things compete for that space: the model weights, the activations or optimizer state produced while computing, and any cache the serving runtime keeps between requests. Training needs far more headroom than serving the same model, because gradients and optimizer state travel alongside the weights.

When a model will not fit on one card, you have two honest options. Shrink it, through reduced precision or quantization, or split it across several cards with tensor or pipeline parallelism. Splitting works, but it turns an interconnect question into your problem, which is the subject of the next section.

How the cards are wired together

Once a job spans more than one GPU, the link between them becomes part of the performance story. Cards can sit on the standard PCI Express bus, or they can be connected by a dedicated high bandwidth fabric on machines designed for that purpose. Provider documentation states which arrangement a given shape uses.

Above the single machine sits the network. Multi node training moves gradients between hosts constantly, so the large platforms offer specialized fabrics for it: Elastic Fabric Adapter on AWS, InfiniBand on the Azure ND series, and RDMA cluster networking on Oracle bare metal shapes. Ordinary VPC networking is fine for serving traffic and unsuitable for tightly coupled training.

The practical rule is to keep a job inside the smallest topology that holds it. One card beats two cards on the same host, and one host beats two hosts, because every boundary you cross adds coordination the framework has to hide.

Getting hold of the capacity in the first place

Accelerated hardware is the most contended category in cloud computing, and that shows up in three ways.

First, regional coverage is uneven. A provider with dozens of locations may offer a given GPU family in only a handful of them, so the region selection logic described in choosing a region and availability zone narrows sharply once accelerators are involved.

Second, quotas for accelerated families often start at zero on a new account and are tracked separately from general purpose compute. Approval for standard vCPUs grants nothing here. The mechanics and the request path are covered in vCPU limits and service quotas, and the request should go in well before the work is scheduled.

Third, having quota is not the same as capacity being free at the moment you launch. Providers sell reservation products for exactly this reason, including capacity reservations on the large platforms and scheduled capacity blocks aimed at training runs. For anything with a deadline, a reservation converts a gamble into a booking.

Drivers, images and the first hour on the box

A bare Linux image does not talk to a GPU. You need a kernel driver, a matching toolkit, and framework builds compiled against that toolkit. Version mismatches between those three layers cause most first day failures.

Two shortcuts remove nearly all of that work. Use the provider's prepared machine image, published under names such as Deep Learning AMI on AWS, the Deep Learning VM images on Google Cloud, or the Data Science images on Azure. Alternatively, run a container built on a vendor base image and let the container runtime expose the device, which is how most teams keep a development laptop and a cloud instance consistent.

The container route pairs naturally with the workflow in running Docker on a cloud server. Whichever you choose, pin versions explicitly, because a silent driver upgrade on rebuild is a genuinely confusing way to lose an afternoon.

Matching the machine to the job

Start from the workload shape rather than the spec sheet. Training and serving pull in opposite directions, a contrast explored in AI training versus inference infrastructure.

• Interactive experimentation: one lighter card, plenty of disk, and a habit of stopping the instance when you walk away.
• Fine tuning a moderate model: a single high memory card, or one host with several cards linked on a fast internal fabric.
• Serving a model to users: the smallest card the model fits on, sized for concurrency rather than peak throughput, as discussed in running LLM inference on cloud GPUs.
• Video encoding or rendering: a family with hardware encoders, which is usually the visualization tier rather than the training tier.

Access to these families comes with the underlying account rather than as a separate purchase, so the platform you settle on matters more than any single instance choice. Teams standardizing on the largest catalog often start from an AWS account, while those who want a short GPU plan list and per hour simplicity look at a Vultr account instead. The AWS and Google Cloud comparison is a useful read when the decision sits between the two largest accelerator catalogs.

Common questions about accelerated instances