Running LLM Inference on Cloud GPUs

Reasons to run the model yourself

Managed model APIs handle most needs, so self hosting should start with a reason. Four come up repeatedly, and they are worth being honest about before committing.

The model is not on any managed catalog, which is common for open weight models and for anything you have fine tuned yourself. Data must stay inside a network boundary you control. You need runtime control that a managed endpoint does not expose, such as custom sampling, unusual context handling or a modified serving stack. Or your volume is high, steady and predictable enough that owning the machines changes the economics.

If none of those apply, a managed endpoint like the ones described in the Amazon Bedrock overview will cost less engineering time. If one of them does, the rest of this article is the shape of the work.

The first question: does the model fit

Everything else is downstream of memory on the accelerator. Three consumers compete for it during serving.

• The weights themselves, whose footprint depends directly on the numeric precision you load them at.
• The key and value cache, which holds intermediate state for every sequence currently in flight and grows with both context length and the number of concurrent requests.
• Working space for the runtime, including activations and any buffers the framework allocates.

The second item is the one people forget. A model that loads comfortably on its own can fail under concurrency, because every additional simultaneous conversation claims more cache. Capacity planning for serving is therefore about concurrency and context length, not about the weights alone.

Two levers change the answer. Reduced precision and quantization shrink the weights, trading some output fidelity for a much smaller footprint, with common approaches operating at eight bit and four bit widths. Tensor parallelism splits the model across several cards on one host, which works well when those cards share a fast internal link, as covered in GPU cloud instances explained.

Choosing a serving runtime

Do not write your own serving loop. A naive script that loads weights and runs one request at a time will leave the accelerator mostly idle, and the gap between that and a real runtime is enormous.

The mature options cluster into a few categories. Dedicated inference servers such as vLLM and Text Generation Inference are built around high throughput token generation and implement the batching and cache management described below. Vendor optimized stacks compile a model for a specific accelerator family, trading portability for efficiency. Lightweight local runners such as Ollama and llama.cpp prioritize ease of setup and run well on modest hardware, including without an accelerator at all.

Most of these expose an HTTP API shaped like the widely used chat completions format. That convention is worth seeking out, because it means your application code is not rewritten when you change runtime or move to a managed endpoint later.

Where throughput actually comes from

Generating tokens one at a time for one user leaves an accelerator badly underused, because the hardware is built for parallel work. Serving runtimes recover that waste in three ways.

Continuous batching merges requests that arrive at different moments into a shared generation step, and lets a finished sequence leave the batch while others continue. Static batching, which waits for a full batch before starting, is far worse for interactive traffic.

Paged cache management stores the key and value cache in fixed blocks rather than one contiguous reservation per sequence. That removes the fragmentation that otherwise forces conservative concurrency limits.

Prefix reuse avoids recomputing the shared beginning of many requests, which matters enormously when every request carries the same long system prompt.

The practical consequence is a trade off you control: raising concurrency raises total throughput and also raises the latency any individual user sees. Pick a target for both, measure them together, and treat a single number as meaningless.

Getting it onto a machine

A workable deployment has four parts, and containerizing them saves repeated pain.

Start from a base image that already carries the accelerator toolkit, install the serving runtime into it, and expose the device to the container through the vendor container runtime. That approach builds on the ordinary workflow in running Docker on a cloud server and keeps a developer machine and a cloud instance behaving identically.

Keep weights out of the image. Model files are large, images should not be, and a rebuild should not mean moving weights again. Store them in object storage and fetch on startup, or keep them on a persistent volume the instance attaches. The patterns in object storage explained cover the first option; the second gives faster restarts at the cost of a resource you must manage.

Pin every version: driver, toolkit, runtime and model revision. Unpinned dependencies in this stack fail in ways that are genuinely hard to diagnose, because the error usually surfaces far from the version that changed.

Scaling, cold starts and queueing

Scaling accelerated serving is harder than scaling web servers, for one reason: a new replica must load many gigabytes of weights before it answers anything. Cold start is measured in minutes, not milliseconds.

That reality shapes the design. Scale on queue depth or on time to first token rather than on processor utilization, which is a poor proxy here. Add capacity earlier than instinct suggests, because the reaction is slow. And keep a warm floor of replicas if your traffic has any interactive component, accepting the idle cost as the price of responsiveness.

In front of the replicas, put a queue with an explicit admission policy. When demand exceeds capacity, a queue that sheds or defers work degrades predictably, while one that accepts everything collapses for everyone. The general mechanics are in autoscaling explained, with the caveat that the timings are an order of magnitude slower here.

Running it day to day

Instrument the numbers that describe the user experience: time to first token, generation rate once streaming starts, queue wait, and the count of requests rejected or timed out. Accelerator utilization and memory occupancy sit alongside those as capacity signals, not as goals in themselves. The habits in monitoring cloud servers transfer directly.

Watch cost with equal care, since an accelerator bills for every allocated hour regardless of traffic. Idle replicas are the largest avoidable line item in most self hosted setups, which is why the discipline in the cost of AI workloads in the cloud is worth adopting on day one rather than after the first invoice.

Where you run this is a real choice. The large platforms offer the widest accelerator catalog and the deepest surrounding services, starting from an AWS account. Teams who want a short plan list and hourly billing often prefer a developer cloud such as a Vultr account or a Scaleway account, and the Hetzner and Vultr comparison gives a sense of how that end of the market differs.

Practical questions