Sign in Get started

DevOps Tutorial · Linux and the Shell · Lesson 10 of 25

Working with JSON and YAML

Read and transform JSON with jq, and write YAML for CI pipelines and Kubernetes without the classic mistakes.

Create a free account to save this page, take notes and track your progress. Sign in

JSON with jq

Cloud CLIs and APIs return JSON; jq extracts what you need:

Filter and group JSON with jq
cat > instances.json <<'JSON'
{"instances": [
  {"id": "i-01", "state": "running", "tags": {"team": "payments"}},
  {"id": "i-02", "state": "stopped", "tags": {"team": "search"}},
  {"id": "i-03", "state": "running", "tags": {"team": "search"}}
]}
JSON
echo "running instances:"
jq -r '.instances[] | select(.state == "running") | .id' instances.json
echo "count per team:"
jq -c '[.instances[].tags.team] | group_by(.) | map({team: .[0], count: length})' instances.json

Output

running instances:
i-01
i-03
count per team:
[{"team":"payments","count":1},{"team":"search","count":2}]

Run this on your computer — see “Get Started” for setup.

Combine with the AWS CLI: aws ec2 describe-instances | jq -r '.Reservations[].Instances[].InstanceId' (the AWS CLI also has its own --query option).

YAML

CI pipelines (GitHub Actions, GitLab CI), Kubernetes manifests, Docker Compose and Ansible use YAML: nested keys by indentation, lists with -.

service: api
replicas: 3
ports:
  - 8080
  - 9090
env:
  LOG_LEVEL: info
  FEATURE_X: "on"      # quote it: unquoted on/off/yes/no can become booleans in YAML 1.1 parsers

Classic YAML mistakes:

  • Indentation with tabs (not allowed) or one space off, which silently changes structure.
  • Unquoted special values: no, on, 08, 1.10 may become booleans, octal-ish numbers or 1.1.
  • Copy-pasted duplicate keys: the last one silently wins in many parsers.
Values YAML changes behind your backTry it yourself
import json

# A YAML 1.1 parser (like PyYAML) reads these unquoted values as:
yaml_11 = {"country": "no", "enabled": "on", "version": "1.10"}
parsed = {"country": False, "enabled": True, "version": 1.1}
for key, text in yaml_11.items():
    print(f"{key}: {text:5} -> {json.dumps(parsed[key])}")
print("quote strings that look like booleans or numbers: version: \"1.10\"")

Output

country: no    -> false
enabled: on    -> true
version: 1.10  -> 1.1
quote strings that look like booleans or numbers: version: "1.10"

Test yourself

Which jq filter prints the id of each item in an array called items?

  1. .items[].id
  2. items.id
  3. $.items.id
  4. .id[]
Show answer

Answer: .items[].id

.items[] iterates the array and .id picks the field from each element.

Was this page helpful?