Observability and Operations · Observability Foundations
Designing Metrics: RED, USE and the Golden Signals
Pick metrics that reveal user impact and saturation without exploding cardinality.
For request-driven services use RED: Rate, Errors, Duration (as histograms for percentiles). For resources use USE: Utilization, Saturation, Errors. Google's four golden signals combine them: latency, traffic, errors, saturation. Add business metrics (orders/min, signups). Keep label cardinality bounded — never label by user ID or raw URL — and use route templates.
# Prometheus queries
sum(rate(http_requests_total{service="checkout"}[5m])) # Rate
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m])) # Error ratio
histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))) # p99 per route
max(pg_pool_waiting_connections) / max(pg_pool_max_connections) # Saturation
Common pitfalls
- Averages instead of percentiles for latency.
- Labels with unbounded values.
- Dashboards with hundreds of graphs and no clear signal.
Key points
- RED for services, USE for resources
- Histograms for percentiles
- Bounded label cardinality
Further reading: Google SRE: Monitoring distributed systems
Practice questions
What are the four golden signals?
Answer
Latency, traffic, errors and saturation — the minimum set to understand a user-facing service's health.
Why is labeling a metric with user_id a problem?
Answer
Each unique value creates a new time series, so millions of users create millions of series, overwhelming storage and queries.
Open in the app to mark lessons complete, take notes and track your progress.