AWS GPU Instances for Inference: How to Choose Compute

The instinct to provision the newest, most powerful GPU instance for an inference workload is usually the wrong instinct, and it’s the most expensive mistake I see teams make on AWS. Using a P5 instance built around H100 GPUs to serve a model that a G6 instance handles comfortably doesn’t buy you better latency at scale, it wastes budget on compute your workload never needed. The right instance for inference is determined by two things: your model’s memory footprint, and your actual latency and concurrency requirement under real traffic, not by which accelerator generation is newest.

Match the Instance Family to the Memory Footprint, Not the Hype

VRAM ceiling is the first filter, and it eliminates most of the decision space immediately. G5 instances, built around the A10G with 24GB of memory, handle smaller models comfortably and lead the mid-range price segment for inference. Once a model’s weights and KV cache push past what a single G5 GPU can hold, the usual next step is G6e, built around the L40S with 48GB, which is the right move specifically when p95 latency under load is your binding constraint after you’ve already tuned batch size and quantization, not before. Past roughly 40 billion parameters, there’s no way around stepping up to P4d, built on A100s with NVLink for tensor-parallel serving, and that family has a hard floor worth knowing before you plan around it: the smallest P4d size is the 24xlarge, eight GPUs, with no smaller option. If you only need one GPU’s worth of capacity for a 30B model, P4d is a poor fit regardless of whether the model needs the memory bandwidth, because you’re paying for eight GPUs to get access to one workload’s worth of capability.

Inferentia2 Is the Cost Lever Most Teams Skip

The Neuron SDK tradeoff

AWS’s own Inferentia2-based Inf2 instances offer the lowest cost per inference on the platform for models the AWS Neuron SDK supports, inf2.xlarge runs roughly 25% cheaper than a comparable g5.xlarge, and the gap widens further for supported transformer architectures at volume. The constraint is real and worth being honest about: Neuron requires a compile step and works best with architectures it’s been tuned for, Llama-family, GPT-NeoX, and BERT-class models specifically. If your model falls outside that supported set, or you’re iterating on architecture choices frequently, the compile and operational overhead isn’t worth it yet. If you’re running a supported architecture at real production volume, where the one-time compile and tooling investment amortizes across millions of inference calls, Inf2 is usually the better cost-per-token outcome than staying on general-purpose NVIDIA GPUs out of inertia.

Two Operational Gotchas That Derail Launch Timelines

GPU quota defaults to zero

New AWS accounts start with a GPU vCPU limit of zero, and requesting a quota increase for P-series or larger G-series instances can take several business days to process. This is a planning-calendar problem, not a technical one, and it’s the kind of detail that turns an otherwise on-schedule launch into a week of waiting because nobody requested the quota increase until the infrastructure was already built and ready to deploy. Request the quota increase the day you decide on an instance family, not the day before you need the capacity.

Spot instances aren’t a serving tier on their own

Spot capacity can cut GPU costs by 60 to 70% and is a legitimate choice for fault-tolerant, checkpointable training or batch inference work. It is not a sound default for primary online inference serving, because interruption risk is incompatible with a latency SLA unless you’ve built an on-demand fallback path that can absorb the loss of a spot instance mid-request without dropping traffic. Treat spot as a cost optimization layered on top of a resilient serving architecture, not a substitute for one.

The Decision Sequence That Actually Works

Size to VRAM first: know your model’s weight footprint plus realistic KV cache growth under your expected context lengths, and let that eliminate instance families before you look at a single price. Check Neuron SDK compatibility second, because if your architecture is supported and your volume justifies the compile overhead, Inferentia2 changes your cost baseline meaningfully. Request GPU quota increases as soon as the instance family is decided, not after the build is finished, since that delay is entirely avoidable and entirely common. Autoscale on GPU utilization and p95 time-to-first-token, not CPU, because CPU rarely bottlenecks a GPU-bound serving workload and autoscaling on the wrong signal means you scale too late or not at all. And pin your driver, CUDA, and inference server versions together using a known-good machine image rather than tracking “latest,” because an untested driver update landing silently in front of a production serving fleet is a self-inflicted outage waiting for a deploy window.

At SIRAYA, the review that catches this most often is a simple one: ask why a given workload is on the instance family it’s on, and see whether the answer is a VRAM or latency requirement, or just a default someone reached for at launch time and never revisited. The teams spending the least per inference aren’t the ones with access to the newest hardware. They’re the ones who sized to the actual workload and kept checking that the sizing still held as the workload changed.