It Works From Our Office

Scenario

Example deployment: an FDE deploys a containerised connector into a bank's Kubernetes cluster. It must call the vendor's cloud API and the bank's internal document system. Everything worked in the vendor's own test environment.

What you see in production

In the bank's cluster the connector fails immediately: calls to the vendor API raise SSL: CERTIFICATE_VERIFY_FAILED, and calls to the telemetry endpoint time out. A developer suggests verify=False to "unblock the pilot". The bank's security team notices that suggestion in a shared channel and pauses the project for a review.

The naive code

import requests

resp = requests.post("https://api.vendor.example/v1/documents", json=payload, timeout=10)
# no proxy configuration, default CA bundle, telemetry hard-coded to telemetry.vendor.example

Reproduce it

Corporate networks add two layers between your code and the internet:

PROXY_ALLOW_LIST = {"api.vendor.example"}               # domains approved by the bank
PROXY_CA = "BankInspectionCA"                            # the proxy re-signs TLS with this CA

def request(host, ca_bundle, via_proxy):
    if not via_proxy:
        return "timeout (direct egress is blocked by the firewall)"
    if host not in PROXY_ALLOW_LIST:
        return "403 from proxy (domain not on the allow-list)"
    if PROXY_CA not in ca_bundle:
        return "CERTIFICATE_VERIFY_FAILED (issuer BankInspectionCA not trusted)"
    return "200 OK"

default_bundle = {"PublicRoots"}
print("default client:       ", request("api.vendor.example", default_bundle, via_proxy=False))
print("with HTTPS_PROXY:     ", request("api.vendor.example", default_bundle, via_proxy=True))
print("+ bank CA in bundle:  ", request("api.vendor.example", default_bundle | {PROXY_CA}, via_proxy=True))
print("telemetry endpoint:   ", request("telemetry.vendor.example", default_bundle | {PROXY_CA}, via_proxy=True))

default client:        timeout (direct egress is blocked by the firewall)
with HTTPS_PROXY:      CERTIFICATE_VERIFY_FAILED (issuer BankInspectionCA not trusted)
+ bank CA in bundle:   200 OK
telemetry endpoint:    403 from proxy (domain not on the allow-list)

Root cause

Enterprises route outbound traffic through proxies that inspect TLS by re-signing certificates with an internal CA, allow only approved destinations, and block direct egress. The connector assumed direct internet access and the public CA bundle, and it called an undocumented telemetry domain. Disabling verification would have removed the protection against man-in-the-middle attacks, which is exactly what a bank's security team must prevent.

The fix

1. Respect standard proxy settings (HTTPS_PROXY, NO_PROXY) in every HTTP client, and make them configurable in the Helm chart.
2. Make the CA bundle configurable: mount the customer's CA certificates and point clients at them (REQUESTS_CA_BUNDLE / SSL_CERT_FILE for Python, NODE_EXTRA_CA_CERTS for Node.js, a truststore for Java).
3. Publish a network requirements list: every domain, port and purpose, with stable endpoints; make optional calls (telemetry) switchable.
4. Offer private connectivity for sensitive customers (PrivateLink, VPN, or self-hosted components).
5. Never disable TLS verification, not even "temporarily".

# values.yaml for the connector
env:
  HTTPS_PROXY: "http://proxy.bank.internal:8080"
  NO_PROXY: ".bank.internal,10.0.0.0/8"
  REQUESTS_CA_BUNDLE: "/etc/ssl/custom/bank-ca-bundle.pem"
telemetry:
  enabled: false

Trade-offs

• TLS inspection means the customer's proxy sees your traffic; for very sensitive data offer private connectivity or certificate pinning exceptions agreed with their security team.
• Configurability adds options to test; ship a connectivity self-check command that verifies each endpoint.
• Allow-listed domains become a contract; changing them needs advance notice.

Detect it in production

• A startup connectivity check that logs which endpoints are reachable and why not (DNS, proxy, TLS, HTTP).
• Error categories in logs (proxy_denied, tls_untrusted, timeout) rather than generic failures.
• An install checklist signed off by the customer's network team.

Key points

• Enterprise networks mean proxies, TLS inspection and allow-lists; design for them.
• Make proxy and CA settings configurable and document every endpoint you need.
• Disabling certificate verification is never the fix.