Monitoring, Logging and Tracing

Logs, metrics and traces

• Logs: timestamped events. Write structured logs (JSON with fields like request_id, user_id, duration_ms) to stdout and let the platform ship them (CloudWatch Logs, Azure Monitor, Cloud Logging, Loki, Elasticsearch/OpenSearch).
• Metrics: numbers over time (request rate, error rate, latency, CPU), cheap to store and alert on (Prometheus, CloudWatch, Datadog).
• Traces: a request's path through services with timings per step (OpenTelemetry, Jaeger, AWS X-Ray).

The golden signals

For every service watch latency, traffic, errors and saturation (how full the resources are).

Use percentiles, not averages

Averages hide the slow requests your users notice. Track p50, p95 and p99.

import random
from statistics import mean, quantiles

random.seed(4)
latencies = [random.gauss(80, 15) for _ in range(980)] + [random.uniform(1500, 3000) for _ in range(20)]
p = quantiles(latencies, n=100)
print(f"average: {mean(latencies):6.0f} ms")
print(f"p50:     {p[49]:6.0f} ms")
print(f"p95:     {p[94]:6.0f} ms")
print(f"p99:     {p[98]:6.0f} ms   <- 1 in 100 users waits this long or longer")

average:    123 ms
p50:         81 ms
p95:        110 ms
p99:       2233 ms   <- 1 in 100 users waits this long or longer