Debugging HTTP Calls from the Command Line

Scenario

A deploy script calls an internal API to register the new version. The deploy reports success, but the service never shows up. Later it hangs for minutes when the API is down. Nobody can tell whether the problem is DNS, the network, or the API.

What you see in production

• Scripts that "succeed" even though the API returned 500.
• Scripts that hang for minutes on unreachable hosts.
• Vague errors like "it can't connect".

The naive code

curl http://registry.internal/api/register -d "version=1.4.0"
echo "registered"

Reproduce it

curl's exit status is 0 for HTTP errors by default, and without --max-time a slow server can hold it for a long time:

cat > api.py <<'EOF'
import http.server, time
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/slow":
            time.sleep(5)
        code = 500 if self.path == "/broken" else 200
        self.send_response(code); self.end_headers(); self.wfile.write(b"response\n")
    def log_message(self, *a): pass
http.server.HTTPServer(("127.0.0.1", 18433), H).serve_forever()
EOF
python3 api.py & server=$!
sleep 0.5
curl -s http://127.0.0.1:18433/broken > /dev/null; echo "HTTP 500, curl exit status: $?"
curl -s http://127.0.0.1:18499/ > /dev/null; echo "nothing listening, curl exit status: $? (connection refused)"
kill $server

HTTP 500, curl exit status: 0
nothing listening, curl exit status: 7 (connection refused)

Root cause

curl treats any HTTP response, including 4xx and 5xx, as a successful transfer, so scripts must check status codes themselves. Without timeouts, an unresponsive server or a dropped connection can block for a long time.

The fix

Fail on HTTP errors, set connect and total timeouts, retry transient failures, and print diagnostics:

cat > api.py <<'EOF'
import http.server, time
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/slow":
            time.sleep(5)
        code = 500 if self.path == "/broken" else 200
        self.send_response(code); self.end_headers(); self.wfile.write(b"response\n")
    def log_message(self, *a): pass
http.server.HTTPServer(("127.0.0.1", 18434), H).serve_forever()
EOF
python3 api.py & server=$!
sleep 0.5
call() {
  curl --fail-with-body --silent --show-error --connect-timeout 2 --max-time 1 \
       --write-out 'status=%{http_code} time=%{time_total}s\n' -o /dev/null "$1" 2>&1 \
    | sed -E 's/time=[0-9.]+s/time=<measured>/'
}
call http://127.0.0.1:18434/ok
call http://127.0.0.1:18434/broken || echo "-> failed with exit status $?"
call http://127.0.0.1:18434/slow || echo "-> failed with exit status $?"
getent hosts localhost
kill $server

status=200 time=<measured>
curl: (22) The requested URL returned error: 500
status=500 time=<measured>
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received
status=000 time=<measured>
::1             localhost

• --fail-with-body (or -f) makes HTTP errors exit non-zero; --show-error prints why.
• --connect-timeout and --max-time bound waiting; --retry 3 --retry-all-errors handles blips (only for idempotent calls).
• -v shows DNS, TLS and headers; --write-out reports status and timings.
• Debug layer by layer: name resolution (getent hosts), TCP (nc -vz host port), TLS (openssl s_client -connect host:443), then HTTP.

Trade-offs

• Retries on non-idempotent POSTs can register things twice; use idempotency keys or don't retry.
• Very short timeouts cause false failures under load; base them on the API's normal latency.
• -k (skip TLS verification) hides real problems; fix certificates instead.

Detect it in production

• Scripts should log the status code and timing of every call.
• Alert on deploy steps that take longer than usual.
• Synthetic checks against internal APIs catch outages before deploys do.

Key points

• curl exits 0 on HTTP errors unless you use --fail.
• Always set connect and total timeouts in scripts.
• Debug DNS, TCP, TLS and HTTP separately.

Further reading: curl man page · Everything curl