Small requests work over a VPN but large responses hang with nothing rejected in the logs. What's the likely cause?

A PMTU black hole: large packets exceed the tunnel's MTU, and the ICMP 'fragmentation needed' messages that would tell the sender to shrink them are blocked, so packets are dropped silently.

What is MSS clamping and where do you apply it?

Rewriting the TCP maximum segment size advertised in SYN packets to fit the path (tunnel MTU minus IP and TCP headers), applied on VPN endpoints, firewalls or routers at the tunnel edge.

What MTUs do you meet in AWS and Google Cloud networks?

AWS: up to 9001 (jumbo frames) inside a VPC, 1500 through an internet gateway, and lower over Site-to-Site VPN. Google Cloud: VPC MTU defaults to 1460 and can be raised; Cloud VPN is lower.

Large Responses Hang Over the VPN — Cloud Networking in Production

Scenario

Example team: a reporting service in an AWS VPC calls an on-premises API over a Site-to-Site VPN. Instances use jumbo frames (MTU 9001) inside the VPC. A security hardening project recently blocked "all ICMP" on the on-premises firewall.

What you see in production

Health checks and small API calls work. Requests that return large responses hang until they time out; curl shows the TLS handshake completing, then nothing. SSH connects, but cat on a big file freezes the session. Nothing is logged as rejected.

The naive code

on-prem firewall:  deny icmp any any        # "ICMP is just ping, block it"
VPN tunnel MTU:    ~1446 bytes (IPsec overhead)
instance MTU:      9001 (jumbo frames inside the VPC)

Reproduce it

Path MTU Discovery relies on routers sending back ICMP "packet too big" when a packet with the don't-fragment flag is too large for the next link:

Path MTU discovery with and without ICMPTry it yourself

PATH = [("VPC", 9001), ("Transit/VPN", 1446), ("on-prem LAN", 1500)]

def send(size, icmp_allowed):
    for hop, mtu in PATH:
        if size > mtu:
            if icmp_allowed:
                return f"{size}-byte packet: ICMP 'packet too big, MTU {mtu}' from {hop} -> sender retries at {mtu}"
            return f"{size}-byte packet: silently dropped at {hop} (ICMP blocked) -> connection hangs"
    return f"{size}-byte packet: delivered"

for icmp in [True, False]:
    print(f"ICMP allowed: {icmp}")
    for size in [600, 1400, 8000]:     # small request, medium, large response segment
        print("  " + send(size, icmp))

mss = 1446 - 20 - 20                   # MSS clamping: tunnel MTU minus IP and TCP headers
print(f"fix: clamp TCP MSS to {mss} on the tunnel so segments never exceed the path MTU")

Output

ICMP allowed: True
  600-byte packet: delivered
  1400-byte packet: delivered
  8000-byte packet: ICMP 'packet too big, MTU 1446' from Transit/VPN -> sender retries at 1446
ICMP allowed: False
  600-byte packet: delivered
  1400-byte packet: delivered
  8000-byte packet: silently dropped at Transit/VPN (ICMP blocked) -> connection hangs
fix: clamp TCP MSS to 1406 on the tunnel so segments never exceed the path MTU

Root cause

Each link has a maximum transmission unit (MTU). TCP sends large segments with the "don't fragment" bit set and lowers its size only when a router replies with ICMP "fragmentation needed" (IPv4) or "packet too big" (ICMPv6). Blocking all ICMP breaks that feedback, so large packets are dropped silently: a PMTU black hole. Small packets (handshakes, small requests) fit, which is why the failure looks selective. Tunnels make it more likely, since IPsec and other encapsulation reduce the usable MTU.

The fix

Allow the ICMP that PMTUD needs: IPv4 type 3 code 4 ("fragmentation needed") and ICMPv6 type 2 ("packet too big"). Never block all ICMPv6, which IPv6 depends on.
Clamp TCP MSS on tunnel endpoints and firewalls so endpoints never send segments larger than the tunnel can carry (for example iptables ... -j TCPMSS --clamp-mss-to-pmtu, or the MSS setting on the VPN device).
Match MTUs where traffic leaves the VPC: AWS supports 9001 inside a VPC, but traffic through an internet gateway, VPN or some peering paths is limited to 1500 or less. Google Cloud VPCs default to MTU 1460, and Cloud VPN to lower (see EC2 network MTU and Google Cloud VPC MTU).
Enable packetization-layer PMTUD on Linux hosts (net.ipv4.tcp_mtu_probing=1), which finds the right size without relying on ICMP.

aws security group (inbound):  ICMP type 3 code 4 ("Destination Unreachable: fragmentation needed") from the VPN range
linux:                         sysctl -w net.ipv4.tcp_mtu_probing=1

Trade-offs

Allowing specific ICMP types is safe and necessary; "block all ICMP" trades a tiny attack surface for hard-to-debug outages.
Lower MTUs everywhere waste bandwidth on headers; jumbo frames inside the VPC help bulk transfers.
MSS clamping only helps TCP; UDP protocols (QUIC, some VPNs) must size packets themselves.

Detect it in production

Test with real sizes: ping -M do -s 1418 host (Linux) finds the largest packet that passes without fragmentation.
Symptom pattern: handshakes and small calls succeed, large transfers stall, nothing is rejected in flow logs.
tracepath host reports the path MTU hop by hop.

Key points

PMTUD needs ICMP "fragmentation needed" / "packet too big"; blocking it creates black holes for large packets.
Tunnels and cloud boundaries change the MTU; jumbo frames stop at the VPC edge.
Allow the specific ICMP types, clamp MSS on tunnels, and test with large packets.