Network Policies That Break DNS

Scenario

Example company: after a penetration test, the team adds a default-deny egress policy to the payments namespace, then allows traffic to the database. Within minutes, the payments API fails every request.

What you see in production

Logs show could not resolve host: db and getaddrinfo EAI_AGAIN. The database is up and reachable by IP. Rolling back the policy fixes everything.

The naive code

kind: NetworkPolicy
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to: [{podSelector: {matchLabels: {app: db}}}]
      ports: [{port: 6379}]
# DNS to kube-system is now blocked

Reproduce it

The policy rules applied to the API pod's two steps, resolving db and connecting to it. (Enforcement depends on the network plugin; this follows the NetworkPolicy semantics that Calico, Cilium and other enforcing plugins implement.)

# NetworkPolicy egress semantics: once any policy with policyTypes: [Egress] selects a pod,
# only connections matching some egress rule of those policies are allowed.
def allowed(pod_labels, dest, port, proto, policies):
    selecting = [p for p in policies if p["select"].items() <= pod_labels.items()]
    if not selecting:
        return True                                   # no egress policy selects the pod: allow all
    for p in selecting:
        for rule in p["egress"]:
            if rule["to"](dest) and (port, proto) in rule["ports"]:
                return True
    return False

coredns = {"labels": {"k8s-app": "kube-dns"}, "namespace": "kube-system"}
db = {"labels": {"app": "db"}, "namespace": "prod-egress"}
api = {"app": "api"}

egress_db_only = {"select": {"app": "api"}, "egress": [
    {"to": lambda d: d["labels"].get("app") == "db", "ports": {(6379, "TCP")}}]}
allow_dns = {"select": {}, "egress": [
    {"to": lambda d: d["namespace"] == "kube-system" and d["labels"].get("k8s-app") == "kube-dns",
     "ports": {(53, "UDP"), (53, "TCP")}}]}

for name, policies in [("no policy", []), ("naive policy", [egress_db_only]), ("plus DNS policy", [egress_db_only, allow_dns])]:
    dns = allowed(api, coredns, 53, "UDP", policies)
    conn = allowed(api, db, 6379, "TCP", policies)
    result = "PONG" if dns and conn else ("failed: could not resolve host db" if not dns else "failed: connection refused")
    print(f"{name:16} DNS query allowed: {dns!s:5}  TCP 6379 to db allowed: {conn!s:5} -> {result}")

no policy        DNS query allowed: True   TCP 6379 to db allowed: True  -> PONG
naive policy     DNS query allowed: False  TCP 6379 to db allowed: True  -> failed: could not resolve host db
plus DNS policy  DNS query allowed: True   TCP 6379 to db allowed: True  -> PONG

Root cause

Once any egress policy selects a pod, all egress not explicitly allowed is denied, including DNS queries to CoreDNS in kube-system. Applications resolve service names before connecting, so they fail even though the destination itself is allowed.

The fix

• Every egress default-deny comes with an allow-DNS policy (UDP and TCP 53 to the cluster DNS pods).
• Write allow rules per dependency (databases, other services, external APIs via CIDR or FQDN policies where the CNI supports them).
• Test policies in a staging namespace with real traffic; tools like the Cilium or Calico policy editors and kubectl probes help.
• Roll out policies gradually and watch drops (CNI flow logs).

Trade-offs

• Strict egress control is valuable against data exfiltration but needs maintenance as dependencies change.
• Standard NetworkPolicies match IPs and pods, not domain names; FQDN rules need CNI-specific policies.
• Policies only work if the CNI enforces them; verify with a test.

Detect it in production

• DNS errors (EAI_AGAIN, NXDOMAIN for in-cluster names) right after policy changes.
• CNI flow logs showing dropped port 53 traffic.
• A CI test that runs a pod under each namespace's policies and checks DNS and key dependencies.

Key points

• Any egress policy turns selected pods into deny-by-default for egress.
• Always allow DNS to the cluster DNS pods alongside egress restrictions.
• Test policies with real probes and watch flow logs during rollout.

Further reading: Network Policies · Network policies in the tutorial