Local NVMe vs EBS: why the 'd' suffix is AWS's best storage deal

Published January 22, 2025

In AWS instance naming, a single letter dictates storage architecture. An m7g.xlarge relies entirely on network-attached EBS volumes for disk I/O. Add a d to the name—m7gd.xlarge—and AWS includes a dedicated 237 GB NVMe SSD directly attached to the physical host via PCIe.

The price difference in us-east-1 is $0.0504 per hour.

For roughly $37 per month, you get raw, direct-attached flash capable of hundreds of thousands of IOPS with sub-100 microsecond latencies. Replicating that performance on Elastic Block Store (EBS) requires expensive provisioned IOPS tiers or multi-volume striping that costs 3x to 35x more.

Despite this, many engineering teams default to EBS-only instances and fight throughput bottlenecks, EBS bandwidth contention, and unpredictable I/O latency. Here is a breakdown of the economics, the performance physics, and the architectural patterns that make instance storage AWS’s most underutilized compute option.

The unit economics: m7g vs m7gd

Let’s look at the baseline Linux on-demand pricing in us-east-1:

The incremental cost for 237 GB of dedicated NVMe flash works out to $0.155 per GB-month ($36.79 / 237 GB).

At first glance, $0.155/GB-month looks higher than baseline gp3 storage ($0.080/GB-month). But baseline gp3 comes with strict limits: 3,000 baseline IOPS and 125 MB/s throughput.

When you factor in the cost of provisioning IOPS and throughput on EBS to match actual NVMe flash performance, the economic comparison flips immediately.

The EBS pricing trap: matching NVMe speeds

Suppose your workload needs 20,000 IOPS and 500 MB/s of sustained throughput for a 237 GB working set.

On gp3, you pay for the base capacity plus provisioned performance ($0.005/IOPS-month above 3,000, and $0.040/MB/s-month above 125 MB/s):

• gp3 base storage: 237 GB × $0.080 = $18.96/mo
• gp3 provisioned IOPS: (20,000 − 3,000) × $0.005 = $85.00/mo (Note: a single gp3 volume caps at 16,000 IOPS, so 20k IOPS requires striping across two volumes)
• gp3 provisioned throughput: (500 − 125) × $0.040 = $15.00/mo
• Total monthly EBS cost: $118.96/month

On io2, Provisioned IOPS SSDs charge $0.125/GB-month for storage and $0.065/IOPS-month for the first 32,000 IOPS:

• io2 base storage: 237 GB × $0.125 = $29.63/mo
• io2 provisioned IOPS: 20,000 × $0.065 = $1,300.00/mo
• Total monthly EBS cost: $1,329.63/month

The local NVMe SSD on m7gd.xlarge is 3.2x cheaper than provisioned gp3 and 36x cheaper than io2, while providing lower latency and higher IOPS than either EBS option can physically deliver to a 4-vCPU instance.

You can inspect global pricing across all volume types in our storage section.

The physics: PCIe bus vs network-attached storage

The cost gap exists because EBS and instance storage operate on fundamentally different hardware paths.

+------------------------------------------------------------------------+
|                          PHYSICAL AWS SERVER                           |
|                                                                        |
|   +-------------------+                     +----------------------+   |
|   |  Graviton3 CPU    |                     | Local NVMe SSD Drive |   |
|   |  (m7gd.xlarge)    | <== PCIe Gen4 ====> | (237 GB hardware)    |   |
|   +-------------------+  (sub-100µs)        +----------------------+   |
|            |                                                           |
|     Nitro Card (PCIe)                                                  |
|            |                                                           |
|            | 12.5 Gbps Network Link                                    |
+------------|-----------------------------------------------------------+
             |
             v (Network packet transit: 800µs to 2ms + jitter)
+------------------------------------------------------------------------+
|                     AWS EBS STORAGE FLEET (gp3/io2)                    |
+------------------------------------------------------------------------+

1. The latency floor

Every read or write to an EBS volume leaves the physical EC2 host, traverses the data center network via the AWS Nitro System, executes on an EBS storage node, and sends an acknowledgment back across the network.

Under ideal conditions, an EBS round-trip takes between 800 microseconds and 2 milliseconds. Under load or cross-rack network congestion, tail latencies (p99/p99.9) regularly spike into tens of milliseconds.

Direct-attached NVMe drives communicate directly over the motherboard’s PCIe bus. I/O requests bypass the network stack entirely, completing in 50 to 90 microseconds with almost zero tail latency jitter.

2. Dedicated EBS bandwidth limits

Every EC2 instance size has an architectural ceiling on how much traffic it can send to EBS, regardless of how many IOPS you pay for on your volumes.

On an m7g.xlarge, AWS allocates:

• Baseline EBS bandwidth: 156.25 MB/s (1,250 Mbps)
• Burst/Max EBS bandwidth: 1,250 MB/s (10,000 Mbps)
• Max EBS IOPS: 40,000

If your instance saturates its baseline EBS throughput, I/O requests queue at the Nitro controller level, adding host-side latency to every read and write. Local NVMe drives do not consume a single bit of your instance’s allocated EBS or network bandwidth.

The catch: the ephemeral lifecycle

The reason AWS prices local NVMe so aggressively is that instance storage is ephemeral. The disk lives in the physical chassis holding your VM.

Understanding when data persists and when it disappears is critical:

When you stop an EC2 instance, AWS releases the physical host. When you start it again, your VM boots on a completely different server in the Availability Zone. The previous host’s NVMe drive is cryptographically wiped and assigned to another tenant.

Because of this, instance storage cannot serve as the primary storage for stateful single-node services without a replication or backup strategy.

Four production architectures for NVMe instance storage

Modern distributed systems are designed to handle node loss. By pairing software-level replication with local NVMe, you get bare-metal disk throughput without sacrificing data safety.

1. Distributed LSM stores & log engines

Databases and streaming platforms built on Log-Structured Merge (LSM) trees or distributed consensus protocols are the natural home for local NVMe.

• Apache Kafka / KRaft: Kafka brokers replicate partition logs across multiple Availability Zones. Putting log segments on local NVMe eliminates disk write stalls and maximizes sequential write throughput. If a broker fails or stops, the partition leader on another node serves traffic while the replaced node resynchronizes from the cluster.
• ScyllaDB / Apache Cassandra: ScyllaDB was explicitly built for shared-nothing, asynchronous direct I/O on NVMe drives. Running ScyllaDB on storage-optimized instances like im4gn.xlarge or compute-paired r7gd.xlarge drops p99 read latencies to sub-millisecond levels at high throughput.
• ClickHouse with S3 object tiering: ClickHouse writes incoming parts to fast local storage, merges them in the background, and offloads frozen or historical parts to Amazon S3 via ClickHouse’s zero-copy S3 storage disks. Local NVMe handles hot ingest and active queries; S3 handles durability and long-term retention.

2. Caching layers & in-memory database spills

Redis, KeyDB, and Memcached workloads often struggle with high memory costs when data sets exceed hundreds of gigabytes.

• Redis on Flash / KeyDB flash tier: Storing 1 TB of data purely in RAM on an r7g.8xlarge costs over $1.70/hr. Using instances with large local NVMe SSDs (such as im4gn or r7gd) allows keeping hot keys in RAM while cold keys reside on NVMe flash. Flash access at 80µs is fast enough for most cache-aside workloads at a fraction of the RAM cost.
• Linux swap buffers for large working sets: For workloads that occasionally exceed allocated RAM during batch jobs, configuring a fast swap file on local NVMe provides a safety buffer that prevents out-of-memory (OOM) killer terminations with minimal performance penalty.

3. CI/CD build runners & container builders

Build systems are purely ephemeral workloads that perform massive amounts of small file I/O (compiling source trees, extracting container layers, running test suites, linking binaries).

• Docker / containerd root directory: Directing /var/lib/docker or /var/lib/containerd to a mounted local NVMe drive drastically speeds up image pulls, layer extraction, and container startup times.
• Compiler artifact caches: Rust (target/), Go (GOCACHE), Node.js (node_modules), and C++ build caches running on local NVMe complete compilations 2x to 4x faster than identical jobs running against gp3 root volumes, reducing total build runner minutes and cloud compute spend.

4. Relational database temporary spaces

You do not have to put your entire PostgreSQL or MySQL database on ephemeral storage to benefit from NVMe.

• PostgreSQL temp_tablespaces: Complex analytical queries, large hash joins, DISTINCT aggregations, and VACUUM operations often spill from work_mem to disk. By mounting a local NVMe drive at /mnt/nvme/pg_temp and setting temp_tablespaces = 'nvme_temp', temporary file creation bypasses EBS entirely. Large sorting operations run at PCIe speeds without consuming primary database EBS IOPS.
• MySQL tmpdir: Directing MySQL’s temporary directory to local NVMe ensures that internal temporary tables created during complex GROUP BY or ORDER BY statements never contend with binary logs or InnoDB data files on EBS.

Instance family selection guide

AWS offers local NVMe across several instance families depending on whether your workload is compute, memory, or storage bound:

For x86_64 workloads, the corresponding c6id, m6id, r6id, and i4i (Intel) or c6ad, m6ad, r6ad, and is4gen (AMD) families follow the same sizing pattern.

Implementation: Automating NVMe initialization

Local NVMe drives arrive unformatted and unmounted every time an instance starts. To use them in production, automate formatting and mounting inside your instance user data script or cloud-init configuration:

#!/usr/bin/env bash
set -euo pipefail

# Identify the local NVMe instance store device
# AWS Nitro NVMe devices expose model info in nvme list or /dev/disk/by-id
NVME_DEVICE=$(lsblk -d -n -o NAME,MODEL | grep -i "Amazon EC2 NVMe Instance Storage" | awk '{print "/dev/" $1}' | head -n 1)

if [ -n "$NVME_DEVICE" ]; then
  # Format with XFS or ext4 if not already formatted
  if ! blkid "$NVME_DEVICE"; then
    mkfs.xfs -f "$NVME_DEVICE"
  fi

  # Create mount point and mount with optimized flags
  mkdir -p /mnt/scratch
  mount -o noatime,nodiratime,discard "$NVME_DEVICE" /mnt/scratch
  chmod 777 /mnt/scratch

  echo "Mounted $NVME_DEVICE at /mnt/scratch"
fi

Using mount options like noatime and nodiratime removes unnecessary filesystem metadata writes, allowing the NVMe controller to deliver maximum raw throughput.

The takeaway

When architecting systems on EC2:

1. Never pay for high EBS IOPS for ephemeral data. Build caches, temporary tables, sort buffers, and swap should run on local NVMe.
2. Cluster-replicated systems belong on d-variants. If Kafka, ScyllaDB, or Elasticsearch already replicates data across 3 Availability Zones, paying for EBS network storage durability is redundant.
3. Check the hourly delta. On modern Graviton instances, upgrading to a d variant costs pennies per hour ($0.05/hr on xlarge) and removes disk I/O bottlenecks from your infrastructure.