Tool cheatsheet
Prometheus and PromQL 3 Cheatsheet
Query metrics with PromQL, and configure scraping, recording rules, alerts and the promtool checker.
Selecting data
- Select a metric — Return the current value of every series with this metric name.
- Filter by labels — Match labels exactly with = and !=, or by regular expression with =~ and !~.
- Select a range of samples — Return the samples from the last five minutes. Range vectors are the input to functions such as rate.
- Look back in time — Compare with the value from an hour ago.
- Find targets that are down — The up metric is 1 when the last scrape worked and 0 when it failed.
Rates and aggregation
- Per-second rate of a counter — Average per-second increase over five minutes. Always use rate (or increase) on counters, never the raw value.
- Total increase over a window — How much a counter grew in the last hour.
- Aggregate across series — Sum the rate across all instances, keeping one series per job and status.
- Error ratio — The share of requests that fail, from two aggregated rates.
- 95th percentile latency — Estimate a percentile from a histogram's buckets. Always sum by le first.
- Top N series — Return the five busiest series.
- Average over time — Smooth a gauge by averaging over a window.
- Predict a trend — Estimate free disk space four hours from now by extrapolating the last six hours.
- Detect a missing metric — Returns 1 when no series match, which is useful for alerting on vanished targets.
- Memory used as a percentage — A common node_exporter query for host memory use.
- CPU used as a percentage — A common node_exporter query for host CPU use.
Configuration, rules and alerts
- Scrape targets — A minimal prometheus.yml that scrapes itself and an application every 15 seconds.
- Precompute an expensive query — A recording rule stores a query's result as a new metric, so dashboards stay fast.
- Define an alert — Fire when the error ratio stays above 5% for ten minutes.
- Route alerts — An Alertmanager configuration that groups alerts and sends pages to one receiver.
Running and checking
- Start Prometheus — Run Prometheus with a config file. The UI is at http://localhost:9090.
- Check the configuration — Validate prometheus.yml and the rule files it references before reloading.
- Check rule files — Validate the syntax of recording and alerting rules.
- Unit-test alert rules — Run tests that supply sample series and check which alerts fire.
- Reload the configuration — Reload without restarting. It needs --web.enable-lifecycle, or send SIGHUP.
- Query over HTTP — Run an instant query against the HTTP API.
- Query from the command line — Run an instant query with promtool.
- List scrape targets — See the health of every target through the API.
Related cheatsheets
- Ansible 2 cheatsheet — Run ad-hoc commands and playbooks, manage inventories, roles, collections and secrets with Ansible.
- Argo CD 3 cheatsheet — Install Argo CD, define Applications declaratively, and manage sync, diffs, history and rollbacks with the argocd CLI.
- AWS CLI 2 cheatsheet — Configure credentials and manage S3, EC2, Lambda, logs, containers and secrets with the AWS CLI v2.
- Azure CLI 2 cheatsheet — Sign in and manage resource groups, VMs, web apps, storage, Kubernetes, registries and secrets with the az command.
- Bash and Linux 5 cheatsheet — Everyday shell and Linux commands for files, searching text, permissions, processes and servers.
- Claude API cheatsheet — Call Claude from Python, TypeScript or curl: messages, streaming, thinking, tools, structured output, caching, batches and error handling.