Kubernetes Tutorial · Operating Kubernetes · Lesson 24 of 26

Troubleshooting Pods

A systematic way to debug Pending, CrashLoopBackOff, ImagePullBackOff and not-ready pods.

Create a free account to save this page, take notes and track your progress. Sign in

The loop

kubectl get pods: what state? (Pending, ImagePullBackOff, CrashLoopBackOff, Running but not ready)
kubectl describe pod NAME: events at the bottom explain scheduling, pulling and probe failures.
kubectl logs NAME (and --previous for the crashed container).
kubectl get events --sort-by=.lastTimestamp for the namespace.
kubectl exec or kubectl debug (ephemeral container) to look inside.

Symptom	Usual causes
Pending	Requests don't fit, node selectors or taints, unbound PVC
ErrImagePull, ImagePullBackOff	Wrong image name or tag, missing registry credentials (Kubernetes alternates the two while retrying)
CrashLoopBackOff	App exits on start: bad config, missing secret, failing migrations
Running, not ready	Readiness probe failing, dependency down, wrong port

CrashLoopBackOff and ImagePullBackOff

kubectl create namespace demo-debug > /dev/null
trap 'kubectl delete namespace demo-debug --wait=false > /dev/null' EXIT
kubectl -n demo-debug run broken --image=busybox:1.37 -- sh -c 'echo "config error: DATABASE_URL is not set"; sleep 1; exit 1' > /dev/null
kubectl -n demo-debug run typo --image=busybox:1.37-doesnotexist > /dev/null
kubectl -n demo-debug wait --for=jsonpath='{.status.containerStatuses[0].state.waiting.reason}'=ImagePullBackOff pod/typo --timeout=180s > /dev/null
kubectl -n demo-debug wait --for=jsonpath='{.status.containerStatuses[0].restartCount}'=2 pod/broken --timeout=180s > /dev/null
kubectl -n demo-debug wait --for=jsonpath='{.status.containerStatuses[0].state.waiting.reason}'=CrashLoopBackOff pod/broken --timeout=180s > /dev/null
kubectl -n demo-debug get pods -o custom-columns=POD:.metadata.name,WAITING:.status.containerStatuses[0].state.waiting.reason,RESTARTS:.status.containerStatuses[0].restartCount --sort-by=.metadata.name | sed -E 's/ [1-9][0-9]*$/ 1+/'
echo "--- logs of the crashed container:"
kubectl -n demo-debug logs broken --previous 2>&1
echo "--- last event for typo:"
kubectl -n demo-debug get events --field-selector involvedObject.name=typo,reason=Failed -o jsonpath='{.items[0].message}' | cut -c1-90; echo

kubectl create namespace demo-debug > /dev/null
trap 'kubectl delete namespace demo-debug --wait=false > /dev/null' EXIT
kubectl -n demo-debug run broken --image=busybox:1.37 -- sh -c 'echo "config error: DATABASE_URL is not set"; sleep 1; exit 1' > /dev/null
kubectl -n demo-debug run typo --image=busybox:1.37-doesnotexist > /dev/null
kubectl -n demo-debug wait --for=jsonpath='{.status.containerStatuses[0].state.waiting.reason}'=ImagePullBackOff pod/typo --timeout=180s > /dev/null
kubectl -n demo-debug wait --for=jsonpath='{.status.containerStatuses[0].restartCount}'=2 pod/broken --timeout=180s > /dev/null
kubectl -n demo-debug wait --for=jsonpath='{.status.containerStatuses[0].state.waiting.reason}'=CrashLoopBackOff pod/broken --timeout=180s > /dev/null
kubectl -n demo-debug get pods -o custom-columns=POD:.metadata.name,WAITING:.status.containerStatuses[0].state.waiting.reason,RESTARTS:.status.containerStatuses[0].restartCount --sort-by=.metadata.name | sed -E 's/ [1-9][0-9]*$/ 1+/'
echo "--- logs of the crashed container:"
kubectl -n demo-debug logs broken --previous 2>&1
echo "--- last event for typo:"
kubectl -n demo-debug get events --field-selector involvedObject.name=typo,reason=Failed -o jsonpath='{.items[0].message}' | cut -c1-90; echo

Output

POD      WAITING            RESTARTS
broken   CrashLoopBackOff   1+
typo     ErrImagePull       0
--- logs of the crashed container:
config error: DATABASE_URL is not set
--- last event for typo:
Failed to pull image "busybox:1.37-doesnotexist": rpc error: code = NotFound desc = failed

Run this on your computer — see “Get Started” for setup.

See Debug Pods.

Test yourself

A pod is in CrashLoopBackOff. What do you check first?

The Service
Its logs (kubectl logs --previous) and describe events
The Ingress
Node disk size

Show answer

Answer: Its logs (kubectl logs --previous) and describe events

The container starts and exits; its own output usually says why.

Was this page helpful?