Choosing a GPU instance on cost, not habit

Published July 7, 2026

The default move when provisioning GPU compute is to pick whatever instance type a tutorial used, or whatever the biggest one in the console is, and hope it’s enough. That’s the same mistake right-sizing with $/vCPU-hr and $/GiB-hr warns against for CPU instances, except the dollar amounts are an order of magnitude higher: the gap between the cheapest and most expensive GPU-hour in our dataset is nearly 50x, and getting the size wrong within a single family can double your bill for the same one GPU.

This guide gives you the two numbers that matter for GPU selection — cost per GPU-hour and GPU memory per dollar — and shows what they look like across every accelerated instance type in our dataset. Look candidates up in the instance explorer, filter to GPU instances, and line up finalists in the comparison tool once you’ve narrowed the field with the method below.

What’s actually in the “GPU” bucket

Filtering our dataset (us-east-1, Linux, on-demand) to instance types with a nonzero GPU count turns up 91 instance types across 27 families. Most carry a real NVIDIA or AMD GPU: g2, g3/g3s, g4ad (AMD Radeon Pro), g4dn, g5, g5g, g6/g6e/g6f, gr6/gr6f, g7/g7e, p2, p3/p3dn, p4d/p4de, p5/p5en, and p6-b200/p6-b300.

A handful aren’t GPUs at all, even though they carry a nonzero count in the same field: inf1 and inf2 are AWS’s own Inferentia inference chips, trn1/trn1n are Trainium training chips, and dl2q runs Qualcomm AI 100 accelerators. Our dataset doesn’t populate a GPU model for any of them, which is your tell — if gpu_model is blank on an instance detail page, you’re looking at a custom AWS or third-party accelerator, not a CUDA-compatible GPU. It also affects compatibility, not just cost: software written against CUDA doesn’t run on Inferentia or Trainium without a compilation step through AWS’s Neuron SDK. This guide focuses on the true-GPU families; treat the accelerator chips as a separate decision with its own software requirements.

$/GPU-hour: the real spread

Divide each instance’s on-demand hourly price by its GPU count and you get cost per GPU-hour — the number that lets you compare a single-GPU g5.xlarge against an 8-GPU p5.48xlarge on equal footing. Here’s the cheapest instance for every GPU model in the dataset, us-east-1, Linux, on-demand:

That’s a 47x spread from the cheapest GPU-hour in the dataset (AMD’s V520 on g4ad.xlarge) to the most expensive (NVIDIA’s B300 on p6-b300.48xlarge). Some of that is real generational capability the dataset can’t measure directly — compute capability climbs from 3.0 on the oldest K520 to 12.0 on the newest Blackwell parts, and gpu_compute_capability on each instance detail page is the one forward-looking number our data actually gives you. We don’t have FLOPS or throughput benchmarks for these chips, so don’t read the price spread alone as a performance spread — it isn’t. What it tells you is that “get the newest GPU to be safe” can mean paying 47x more per GPU-hour than a workload that would run fine on a T4 or an A10G.

GPU memory is usually the binding constraint

Compute capability is a nice-to-have. Memory is a hard limit. A model or dataset that needs 40 GB of GPU memory doesn’t run on a 24 GB card no matter how much cheaper that card’s GPU-hour is — it fails to load, full stop. That makes $/GB of GPU memory per hour at least as important as $/GPU-hour for ML training and inference workloads, and it produces a different ranking:

Two things stand out. First, the newest, most expensive-per-GPU-hour instances aren’t the worst deal on memory — the massive HBM pools on the B200 and B300 mean their $/GB-hr lands close to the A100 and H100, not far above it. Second, and more useful in practice: the V100 on p3.2xlarge, at a middling $3.06/GPU-hr, has the worst memory economics in this table by a wide margin, because it’s carrying only 16 GiB. A workload that’s memory-bound rather than compute-bound (most training and many inference workloads are) pays for that 16 GiB ceiling regardless of how the raw GPU-hour price compares. Check both numbers before assuming the cheaper sticker price is the cheaper instance for your workload.

One inconsistency in the data, stated plainly rather than smoothed over: p5en.48xlarge’s GPU memory field reads 141 GB, which is the H200’s per-GPU spec, not the roughly 1,128 GB you’d expect across all 8 GPUs the way p4d and p6 report theirs. We’re passing that number through as recorded rather than guessing at a correction — check the instance detail page yourself before sizing a p5en deployment on GPU memory alone.

Same family, wildly different economics per GPU

Multi-GPU instances complicate the picture further, because within some families, sizing up doesn’t mean getting more GPUs — it means paying more for the same one GPU with extra vCPUs and RAM attached. Compare every size in the g5 family (NVIDIA A10G, us-east-1, Linux, on-demand):

g5.16xlarge still has exactly one A10G, the same GPU as g5.xlarge, but costs over 4x as much per GPU-hour because AWS bolted on far more vCPU, RAM, and local NVMe around it. If your job is GPU-bound and doesn’t need that surrounding hardware, g5.16xlarge is close to the worst way to buy an A10G in this family. The 4-GPU and 8-GPU sizes reset to a better per-GPU rate than the single-GPU jumbo sizes, though they’re still pricier per GPU than the plain xlarge. The same shape shows up in g6, g6e, g7, and g7e — check before you size up within any of these families, the same way right-sizing with $/vCPU-hr tells you to check within CPU families.

The p3 and p5 training families don’t do this. Their per-GPU price is flat across every size:

A p3.16xlarge costs exactly 8x a p3.2xlarge, unit for unit, the same linear scaling the right-sizing guide found in CPU families like m7i. Confirm that flat scaling directly on the comparison tool before you commit to a family — it isn’t universal across GPU instance types the way it is across most CPU ones.

Going wide vs. going big

Once you know your $/GPU-hour and $/GB-hr, the last decision is shape: many small instances, or one large multi-GPU instance.

For embarrassingly parallel work — batch inference over a queue of independent jobs, rendering frames, running many independent training trials for hyperparameter search — going wide is usually the cheaper and more flexible option. A fleet of g5.xlarge or g6.xlarge instances that scales with queue depth has no reason to consolidate onto a single big multi-GPU box: each job needs one GPU and doesn’t talk to the others, so buying 8 separate GPUs across 8 cheap instances costs about the same as 8 GPUs in one g5.48xlarge (compare the $/GPU-hr rows above) while giving you finer-grained autoscaling and no single point of failure.

Single-node multi-GPU instances earn their keep when the GPUs need to talk to each other fast and often — training a model too large for one GPU’s memory, or any job that shards computation across GPUs within a training step. That kind of workload depends on high-bandwidth interconnects like NVLink between GPUs on the same node, which is a real hardware difference between instance types but isn’t something our pricing dataset captures directly; check AWS’s own instance documentation for the interconnect topology of a specific multi-GPU type before committing to it for large-model training. What the pricing data does tell you is the shape of the tradeoff: going wide is a real, cheaper option whenever your workload doesn’t need that interconnect, and the flat per-GPU pricing on p3 and p5 (above) means there’s no cost penalty for picking the multi-GPU size once you’ve confirmed you actually need it.

If you only need a slice of a GPU rather than a whole one, check the g6f and gr6f families before renting a full g6.xlarge. They carve a single L4 into fractional shares — g6f.large runs 1/8 of an L4’s compute with 3 GB of its memory for $0.202/hr, well under the $0.8048/hr of a whole g6.xlarge. That’s a real option for small inference workloads that would otherwise waste most of a full GPU, though it isn’t free: the fractional sizes here work out to a higher effective $/GPU-hr than just buying the whole card, so only reach for them when your workload genuinely doesn’t need a full GPU’s memory or compute.

How to decide

1. Establish your GPU memory floor first. Whatever you’re running — a model checkpoint, a batch size, a rendering scene — has a minimum GPU memory requirement below which it simply doesn’t run. That number rules out every instance below it before price enters the conversation.
2. List every instance type that clears the memory floor, using the instance explorer filtered to GPU instances, and note the GPU model and count for each.
3. Compute $/GPU-hr and $/GB-hr for the shortlist using the on-demand price from each instance detail page divided by GPU count and GPU memory. Weight whichever metric matches your bottleneck: compute-bound work cares more about $/GPU-hr, memory-bound work (most training, much inference) cares more about $/GB-hr.
4. Check whether sizing up within a family gets you more GPUs or just more CPU/RAM around the same GPU count, the way the g5 table above shows. If it’s the latter, a smaller size plus horizontal scaling is probably the better deal.
5. Decide wide vs. big based on whether your GPUs need to talk to each other. Independent jobs go wide on cheap single-GPU instances. Training runs that shard a model or a batch across GPUs need a genuine multi-GPU node — confirm the interconnect it offers before committing.
6. Confirm the shortlist side by side in the comparison tool, and check the family page (e.g. families/g6 or families/p5) for generation and use-case notes before finalizing.

Making the call

Small inference or dev/test workloads, memory needs under ~24 GB: g4dn, g5, or g6 at the smallest size that fits, or a fractional g6f/gr6f slice if you don’t need a whole card. These sit at the cheap end of $/GPU-hr in our data and cost the least to experiment with.

Larger models or batches that need real headroom, still on one GPU: g6e (48 GB, L40S) or g7e (96 GB, RTX PRO 6000 Blackwell) before jumping to a multi-GPU p-family instance you don’t need yet. Check the memory floor from step 1 against these before assuming you need eight GPUs.

Distributed training or a model too large for one GPU’s memory: p3, p4d/p4de, p5/p5en, or p6-b200/p6-b300, sized to the number of GPUs your sharding strategy actually uses — their flat per-GPU pricing means there’s no size penalty for picking the right GPU count.

Many independent, embarrassingly parallel jobs: go wide on the cheapest single-GPU instance that clears your memory floor rather than consolidating onto one large multi-GPU box. The per-GPU cost is close to identical either way, and going wide gives you finer autoscaling.

Bottom line: the newest, biggest GPU instance is rarely the right default, the same way the biggest CPU instance rarely is. Our dataset shows a 47x spread in $/GPU-hour and a real, separate spread in $/GB-hr that doesn’t move in lockstep with it — the V100 on p3 is a mediocre memory deal despite a mid-range GPU-hour price, and the newest Blackwell parts are a better memory deal than their sticker price suggests. Set your memory floor first, then let $/GPU-hr and $/GB-hr, not generation or habit, pick the instance.