Why Is Our Bill Mostly Networking?

Scenario

Example team: a microservices platform on EKS across three availability zones. Services call each other about 40,000 times per second, logs and metrics ship to a SaaS vendor, nightly exports go to a partner, and container images are pulled from a public registry through NAT gateways.

What you see in production

The bill shows data transfer and NAT gateway charges larger than the EC2 cost of the cluster. The team assumed networking inside AWS was free.

The naive code

service A (zone a) -> service B pods spread over zones a, b, c   (2/3 of calls cross zones)
pods (private subnets) -> NAT gateway -> public image registry, log SaaS, S3

Reproduce it

Data transfer has many small per-GB prices that add up at volume:

PRICES = {"cross-AZ (both directions)": 0.02, "NAT processing": 0.045, "internet egress": 0.09, "S3 via gateway endpoint": 0.0}
traffic_gb = {
    "cross-AZ (both directions)": 180_000 * 2 / 3,   # service-to-service, 2/3 of calls leave the zone
    "NAT processing": 60_000,                        # logs to SaaS, image pulls, S3 through NAT
    "internet egress": 25_000,                       # logs SaaS + partner exports
}
total = 0
for item, gb in traffic_gb.items():
    cost = gb * PRICES[item]
    total += cost
    print(f"{item:28} {gb:9,.0f} GB  ${cost:9,.0f}")
print(f"{'total':28} {'':9}     ${total:9,.0f} per month")

saved = traffic_gb["cross-AZ (both directions)"] * 0.7 * PRICES["cross-AZ (both directions)"] \
      + 35_000 * PRICES["NAT processing"]            # keep most calls in-zone; S3 + ECR via endpoints
print(f"after zone-aware routing and VPC endpoints: about ${total - saved:,.0f} per month")

cross-AZ (both directions)     120,000 GB  $    2,400
NAT processing                  60,000 GB  $    2,700
internet egress                 25,000 GB  $    2,250
total                                      $    7,350 per month
after zone-aware routing and VPC endpoints: about $4,095 per month

Root cause

Inside a region, traffic between availability zones is billed per GB in each direction; traffic through a NAT gateway pays a per-GB processing charge on top of anything else; and traffic out to the internet is the most expensive. Spreading every service evenly across zones (good for availability) without zone-aware routing makes most calls cross zones. Nobody measured traffic by path.

The fix

1. Measure: VPC Flow Logs plus Cost and Usage Report line items to find top talkers by path.
2. Keep calls in-zone where safe: Kubernetes topology-aware routing / traffic distribution preferring same-zone endpoints, zone-aware load balancing; but keep multi-AZ deployments for availability.
3. Avoid NAT for cloud services: gateway endpoints for S3/DynamoDB (free), interface endpoints for ECR, CloudWatch and STS; pull-through caches for public images.
4. Reduce egress: compress, batch and sample telemetry; ship logs with a private connection where the vendor offers one; cache content at the CDN.
5. Architecture: co-locate chatty services; avoid cross-region replication you don't need.

Trade-offs

• Strict same-zone routing can overload one zone's pods; use "prefer same zone" with fallback, and enough replicas per zone.
• Interface endpoints have hourly costs per zone; they pay off above a modest traffic volume.
• Sampling telemetry saves money but loses detail; keep errors and traces of slow requests.

Detect it in production

• Cost explorer views by usage type (DataTransfer-Regional-Bytes, NatGateway-Bytes, DataTransfer-Out-Bytes).
• Flow-log dashboards of bytes by source, destination and zone.
• Alerts when networking exceeds an agreed share of the workload's cost.

Key points

• Cross-AZ, NAT processing and internet egress are all billed per GB.
• Measure by path, keep chatty traffic in-zone, and use VPC endpoints instead of NAT for cloud services.
• Reduce telemetry and egress volume before buying more capacity.