Cross-AZ data transfer: the silent tollbooth in multi-AZ architectures

Published August 4, 2025

AWS Well-Architected guidelines recommend deploying workloads across multiple Availability Zones (AZs) for resilience. If an entire data center facility loses power or network connectivity, your application continues running in the surviving zones.

What the reference architectures often skip is the networking cost: moving data across AZ boundaries inside the same region is not free. AWS meters inter-AZ traffic at $0.01 per GB sent and $0.01 per GB received, creating a $0.02 per GB total round-trip tax on internal communication.

For monolithic applications with light database traffic, this fee is negligible. But in modern distributed environments—Kubernetes clusters running hundreds of microservices, multi-node Kafka pipelines, Elasticsearch clusters, and caching layers—cross-AZ data transfer often becomes one of the largest single lines on the AWS bill, generating thousands of dollars in monthly costs without serving a single byte of external internet egress.

Here is how cross-AZ billing works, where the hidden terabytes accumulate, and four practical engineering fixes to cut internal data transfer costs.

The mechanics of cross-AZ data transfer pricing

According to AWS networking data in our networking section, data transfer within a VPC falls into three pricing categories:

Same AZ:
[ EC2 Instance A (AZ-1a) ] -------- (Free: $0.00/GB) --------> [ EC2 Instance B (AZ-1a) ]

Cross AZ:
[ EC2 Instance A (AZ-1a) ] -- $0.01/GB egress --> [ AZ Boundary ] -- $0.01/GB ingress --> [ EC2 Instance B (AZ-1b) ]
Total cost for point-to-point payload: $0.02/GB ($20 per TB)

Within the same AZ, private IP traffic costs nothing. Once a packet crosses the physical boundary to another AZ:

1. The sending instance’s account is billed $0.01/GB for Inter-AZ Data Transfer Out.
2. The receiving instance’s account is billed $0.01/GB for Inter-AZ Data Transfer In.

If Service A in us-east-1a sends a 1 GB payload to Service B in us-east-1b, and Service B returns a 1 GB response, you pay:

• 1 GB egress from AZ-1a ($0.01) + 1 GB ingress to AZ-1b ($0.01) = $0.02
• 1 GB egress from AZ-1b ($0.01) + 1 GB ingress to AZ-1a ($0.01) = $0.02
• Total round-trip cost: $0.04 for 2 GB of total bi-directional wire transfer.

At scale, 50 TB of monthly cross-AZ traffic results in $1,000/month in pure internal routing overhead.

Monthly Cross-AZ Cost Scaling:
  10 TB cross-AZ payload  = $200 / month
  50 TB cross-AZ payload  = $1,000 / month
  100 TB cross-AZ payload = $2,000 / month
  500 TB cross-AZ payload = $10,000 / month

Where the terabytes vanish: three common culprits

1. Distributed datastores (Kafka, Cassandra, Elasticsearch)

Distributed storage systems replicate writes across nodes to guarantee fault tolerance. When nodes are distributed across three AZs for high availability, every write replicates across the AZ perimeter.

Consider an Apache Kafka cluster ingesting log and metrics streams:

• Ingestion rate: 100 TB/month of raw event data.
• Replication factor: 3 (1 leader, 2 followers), balanced across 3 AZs (us-east-1a, us-east-1b, us-east-1c).
• Producer in us-east-1a writes to the partition leader in us-east-1a.

Producer (AZ-a) 
   │
   ▼ (Local write: $0.00)
Leader Broker (AZ-a)
   ├─── Replicate to Follower (AZ-b) ───> 100 TB @ $0.01 egress + $0.01 ingress = $1,000
   └─── Replicate to Follower (AZ-c) ───> 100 TB @ $0.01 egress + $0.01 ingress = $1,000

Total Replication Cost: 200 TB cross-AZ traffic = $2,000/month

The 100 TB ingestion generates 200 TB of cross-AZ replication traffic ($2,000/month).

If downstream consumer applications running in us-east-1b and us-east-1c consume this data directly from the leader in us-east-1a without follower-fetching, that adds another 100 TB ($1,000/month) of cross-AZ consumer reads. A single pipeline moving 100 TB of application events can easily generate $3,000/month in inter-AZ network fees.

2. Kubernetes / EKS cross-AZ service meshes

In a standard Amazon EKS cluster spanning 3 AZs, Kubernetes assigns pods randomly across worker nodes unless configured with explicit topology constraints.

When Service A calls Service B through a standard ClusterIP Service or an internal Application Load Balancer (ALB), kube-proxy distributes connections uniformly via round-robin across all healthy endpoint pods regardless of zone.

In a 3-AZ cluster with evenly distributed pods:
- 33.3% of requests land on a pod in the SAME AZ (Free: $0.00)
- 66.7% of requests land on a pod in a DIFFERENT AZ ($0.02/GB)

In a microservice call chain where Request $\rightarrow$ Service A $\rightarrow$ Service B $\rightarrow$ Service C $\rightarrow$ Database:

• Hop 1 (A $\rightarrow$ B): 66.7% chance of crossing AZs
• Hop 2 (B $\rightarrow$ C): 66.7% chance of crossing AZs
• Hop 3 (C $\rightarrow$ DB): 66.7% chance of crossing AZs

Over millions of daily RPC requests transmitting JSON or gRPC payloads, two-thirds of your internal cluster bandwidth is billed at cross-AZ rates.

3. Remote cache lookups and chatty state stores

A common anti-pattern is deploying an in-memory caching tier (Redis or Memcached) as a single primary node in us-east-1a while application compute instances run in us-east-1a, us-east-1b, and us-east-1c using instances like c7g.xlarge.

If the application executes 20 cache queries per user request to assemble session data, permissions, and feature flags:

• 66.7% of all Redis reads cross AZ boundaries.
• The application incurs both the financial data transfer charge and an added 1–2 ms of round-trip network latency per cross-AZ query compared to sub-millisecond local AZ access.

Four architectural fixes to eliminate cross-AZ waste

1. Kubernetes Topology Aware Routing

Kubernetes provides native support for keeping network traffic within the originating zone. Since Kubernetes 1.27+, Topology Aware Routing (enabled via service.kubernetes.io/topology-mode: Auto) instructs kube-proxy or the CNI plugin to prefer endpoints in the same zone as the calling client.

apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: production
  annotations:
    service.kubernetes.io/topology-mode: Auto
spec:
  selector:
    app: order-service
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

How it works:

1. The EndpointSlice controller checks node labels (specifically topology.kubernetes.io/zone).
2. It assigns endpoint subsets to each zone proportionally based on the allocatable vCPU capacity in that zone.
3. When a pod in us-east-1a sends traffic to order-service, kube-proxy routes the request exclusively to pods running in us-east-1a.
4. Cross-AZ routing only occurs during failover if no healthy local endpoints exist.

Enabling this annotation drops internal Kubernetes microservice cross-AZ traffic from ~67% down to near 0%.

2. AZ-aware Kafka routing (KIP-392 Fetch from Follower)

Historically, Kafka consumers could only read data from the partition leader broker. If the leader was in us-east-1a, a consumer in us-east-1b had to pull every message across the AZ boundary.

With Kafka 2.4+ and KIP-392, consumers can fetch messages directly from the closest in-sync replica (ISR) follower in their own AZ.

# 1. Broker configuration (server.properties)
# Map each broker to its AWS Availability Zone ID or Name
broker.rack=us-east-1a
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector

# 2. Consumer client configuration
# Configure consumer instances with their local zone ID
client.rack=us-east-1a

When configured:

• Producers still write to the partition leader (incurring cross-AZ replication costs for durability).
• Consumers in us-east-1a, us-east-1b, and us-east-1c read directly from the follower broker residing in their respective local zone.
• Eliminates 100% of consumer cross-AZ data transfer fees.

3. Co-locating chatty services with zone affinity

For high-throughput, low-latency microservice architectures, group services into independent, self-contained zone pools rather than letting calls cross AZ boundaries randomly.

Zone-Isolated Architecture:

AZ-1a: [ Ingress ALB ] ──> [ Web Pod ] ──> [ API Pod ] ──> [ Local Redis Replica ]
                                                                     ▲
                                                                     │ Async replication
AZ-1b: [ Ingress ALB ] ──> [ Web Pod ] ──> [ API Pod ] ──> [ Redis Primary (Leader) ]

Implementation guidelines:

• Local Ingress Routing: Configure your external load balancer (ALB/NLB) with Cross-Zone Load Balancing disabled (load_balancing.cross_zone.enabled = false) if each AZ has equal capacity. Ingress traffic stays in the zone where DNS routed it.
• Local Read Caching: Deploy Redis read replicas into each AZ. Route all read queries to localhost or the local AZ Redis endpoint, sending only cache writes to the primary node in the designated leader zone.
• Strict Multi-AZ for Storage: Reserve cross-AZ network bandwidth for asynchronous database replication (e.g., Aurora primary $\rightarrow$ Aurora replica) rather than synchronous operational RPCs.

4. Identifying top talkers with VPC Flow Logs and Contributor Insights

To locate which services are driving cross-AZ data transfer in your AWS environment, enable VPC Flow Logs with custom format fields that capture subnet and AZ metadata.

Recommended VPC Flow Log Format:

${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${subnet-id} ${az-id} ${pkt-src-aws-service} ${pkt-dst-aws-service}

CloudWatch Logs Insights Query:

Run this query in CloudWatch Logs Insights over your VPC Flow Log group to aggregate total transfer bytes between distinct subnets:

filter srcaddr not like /^169\.254\./ and dstaddr not like /^169\.254\./
| stats sum(bytes) as TotalBytes by srcaddr, dstaddr, subnetId
| sort TotalBytes desc
| limit 25

CloudWatch Contributor Insights Rule:

Create a Contributor Insights rule on VPC Flow Logs to produce real-time rankings of top IP pairs crossing AZs:

{
  "AggregateOn": "Sum",
  "Contribution": {
    "Filters": [
      {
        "Match": "action",
        "EqualTo": "ACCEPT"
      }
    ],
    "Keys": [
      "srcaddr",
      "dstaddr"
    ],
    "ValueOf": "bytes"
  },
  "LogFormat": "CLF",
  "LogGroupName": "/aws/vpc/production-flow-logs"
}

This report highlights which pod IPs, database endpoints, or broker instances generate the highest inter-AZ network volume. You can cross-reference instance families and compute sizing in the instance explorer to balance node densities per AZ.

Summary checklist

Audit your AWS bill for the DataTransfer-Regional-Bytes line item. If your monthly regional data transfer exceeds a few hundred dollars, cross-AZ traffic is the primary driver—and fixing it requires routing adjustments rather than tearing down your multi-AZ reliability.