DevOps in Production · Releases You Can Undo
The Key in the Image
Deleting a secret in a later Dockerfile step doesn't remove it from earlier layers. Use build secrets, .dockerignore and scanning, and rotate anything exposed.
Scenario
Example team: a service's Dockerfile copies the whole repository, including a local .env file with a cloud access key used to download private packages, then deletes .env in a later step "for security". The image is pushed to a registry that was accidentally made public.
What you see in production
Three weeks later the cloud bill shows crypto-mining instances in an unused region. The access key from that .env file was used to create them. The final image's file system has no .env, so the team initially doesn't believe the image was the source.
The naive code
FROM node:22-slim
WORKDIR /app
COPY . . # includes .env with AWS_ACCESS_KEY_ID / SECRET
RUN export $(cat .env) && npm ci # private registry token and cloud key used here
RUN rm .env # "removed" -- only in this new layer
CMD ["node", "server.js"]
Reproduce it
An image is a stack of layers; each one records the files added or deleted by one instruction. Deleting a file adds a "whiteout" in a new layer, but the earlier layer still contains the file, and anyone who pulls the image can extract it:
layers = [
("FROM node:22-slim", {"add": {"/usr/local/bin/node": "..."}}),
("COPY . .", {"add": {"/app/server.js": "...", "/app/.env": "AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI..."}}),
("RUN npm ci", {"add": {"/app/node_modules/": "..."}}),
("RUN rm .env", {"delete": ["/app/.env"]}),
]
final = {}
for _, change in layers:
final.update(change.get("add", {}))
for path in change.get("delete", []):
final.pop(path, None)
print("final file system has .env:", "/app/.env" in final)
for i, (instruction, change) in enumerate(layers):
for path, content in change.get("add", {}).items():
if ".env" in path:
print(f"layer {i} ({instruction}) still contains {path}: {content[:28]}...")
Output
final file system has .env: False layer 1 (COPY . .) still contains /app/.env: AWS_SECRET_ACCESS_KEY=wJalrX...
Root cause
Images are append-only stacks of layers, and registries serve every layer. docker history, docker save or any layer-extraction tool reveals files that later layers deleted. Copying the whole build context (no .dockerignore) brought local secrets into the image in the first place, and a long-lived access key turned a leak into an incident.
The fix
- Rotate the leaked key immediately and check audit logs (CloudTrail) for its use; removing the image doesn't un-leak it.
- Never put secrets in the build context: a
.dockerignorethat excludes.env*,.git, keys and local config. - Use build secrets that are mounted only for one
RUNstep and never stored in a layer:
# syntax=docker/dockerfile:1
FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci --omit=dev
COPY src/ ./src/
USER node
CMD ["node", "src/server.js"]
docker build --secret id=npmrc,src=$HOME/.npmrc -t api:$(git rev-parse --short HEAD) .
- No long-lived keys for builds: CI authenticates with OIDC and short-lived tokens; runtime uses workload identity.
- Scan images and repositories for secrets (GitHub secret scanning, gitleaks, trufflehog) in CI, and keep registries private by default.
Trade-offs
- BuildKit secret mounts need BuildKit (default in current Docker) and changes to build scripts.
- Multi-stage builds also help (secrets used only in a discarded build stage), but a secret in a stage that's pushed or cached remotely can still leak; secret mounts are safer.
- Scanners have false positives; tune them rather than turning them off.
Detect it in production
- Registry and secret-scanning alerts on new images;
docker history --no-truncchecks in CI. - Cloud audit logs for access-key use from unexpected IPs or regions; billing anomaly alerts.
- Periodic review of public repositories and registries.
Key points
- A file deleted in a later layer is still in the image.
- Keep secrets out of the build context, mount them with BuildKit secrets, and prefer short-lived credentials.
- A leaked secret must be rotated; deleting it isn't enough.
Practice questions
Why doesn't `RUN rm .env` remove a secret from a Docker image?
Answer
Each instruction creates a layer. The file stays in the layer that added it; the deletion only hides it in later layers. Anyone with the image can extract earlier layers.
How do you use a secret during a Docker build without storing it in the image?
Answer
BuildKit secret mounts: RUN --mount=type=secret,id=... exposes the secret only to that step, and docker build --secret id=...,src=... provides it. Exclude secrets from the context with .dockerignore.
A secret was pushed in an image to a public registry. What do you do?
Answer
Rotate or revoke it immediately, check audit logs for use, remove the image, then fix the build (secret mounts, .dockerignore) and add secret scanning so it can't recur.
Open in the app to mark lessons complete, take notes and track your progress.