Debug a distroless pod with an ephemeral container

Approach

kubectl debug adds an ephemeral container (with your tools) to the running pod, sharing the target container's process namespace, so you can see its processes and filesystem via /proc.

Solution

kubectl debug -it pod/api-7d9f8 --image=busybox:1.37 --target=api -- sh
# inside: ps; ls /proc/1/root/app; wget -qO- localhost:8080/ready
kubectl debug node/worker-3 -it --image=busybox:1.37        # a shell on a node (host filesystem at /host)
kubectl debug pod/api-7d9f8 -it --copy-to=api-debug --container=api --image=ghcr.io/acme/api:1.4.2-debug -- sh

Pitfalls

• Debug images pulled from public registries in locked-down clusters; mirror one internally.
• Leaving copied debug pods (--copy-to) running.

Variations

• --profile=sysadmin or netadmin for privileged debugging when policy allows.
• Use kubectl cp to copy files out for local analysis.

Further reading: Debugging with an ephemeral container