DevOps in Production · Releases You Can Undo

The Key in the Image

Deleting a secret in a later Dockerfile step doesn't remove it from earlier layers. Use build secrets, .dockerignore and scanning, and rotate anything exposed.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

Example team: a service's Dockerfile copies the whole repository, including a local .env file with a cloud access key used to download private packages, then deletes .env in a later step "for security". The image is pushed to a registry that was accidentally made public.

What you see in production

Three weeks later the cloud bill shows crypto-mining instances in an unused region. The access key from that .env file was used to create them. The final image's file system has no .env, so the team initially doesn't believe the image was the source.

The naive code

FROM node:22-slim
WORKDIR /app
COPY . .                                  # includes .env with AWS_ACCESS_KEY_ID / SECRET
RUN export $(cat .env) && npm ci          # private registry token and cloud key used here
RUN rm .env                               # "removed" -- only in this new layer
CMD ["node", "server.js"]

Reproduce it

An image is a stack of layers; each one records the files added or deleted by one instruction. Deleting a file adds a "whiteout" in a new layer, but the earlier layer still contains the file, and anyone who pulls the image can extract it:

Layers remember everythingTry it yourself
layers = [
    ("FROM node:22-slim", {"add": {"/usr/local/bin/node": "..."}}),
    ("COPY . .", {"add": {"/app/server.js": "...", "/app/.env": "AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI..."}}),
    ("RUN npm ci", {"add": {"/app/node_modules/": "..."}}),
    ("RUN rm .env", {"delete": ["/app/.env"]}),
]

final = {}
for _, change in layers:
    final.update(change.get("add", {}))
    for path in change.get("delete", []):
        final.pop(path, None)
print("final file system has .env:", "/app/.env" in final)

for i, (instruction, change) in enumerate(layers):
    for path, content in change.get("add", {}).items():
        if ".env" in path:
            print(f"layer {i} ({instruction}) still contains {path}: {content[:28]}...")

Output

final file system has .env: False
layer 1 (COPY . .) still contains /app/.env: AWS_SECRET_ACCESS_KEY=wJalrX...

Root cause

Images are append-only stacks of layers, and registries serve every layer. docker history, docker save or any layer-extraction tool reveals files that later layers deleted. Copying the whole build context (no .dockerignore) brought local secrets into the image in the first place, and a long-lived access key turned a leak into an incident.

The fix

  1. Rotate the leaked key immediately and check audit logs (CloudTrail) for its use; removing the image doesn't un-leak it.
  2. Never put secrets in the build context: a .dockerignore that excludes .env*, .git, keys and local config.
  3. Use build secrets that are mounted only for one RUN step and never stored in a layer:
# syntax=docker/dockerfile:1
FROM node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci --omit=dev
COPY src/ ./src/
USER node
CMD ["node", "src/server.js"]
docker build --secret id=npmrc,src=$HOME/.npmrc -t api:$(git rev-parse --short HEAD) .
  1. No long-lived keys for builds: CI authenticates with OIDC and short-lived tokens; runtime uses workload identity.
  2. Scan images and repositories for secrets (GitHub secret scanning, gitleaks, trufflehog) in CI, and keep registries private by default.

Trade-offs

  • BuildKit secret mounts need BuildKit (default in current Docker) and changes to build scripts.
  • Multi-stage builds also help (secrets used only in a discarded build stage), but a secret in a stage that's pushed or cached remotely can still leak; secret mounts are safer.
  • Scanners have false positives; tune them rather than turning them off.

Detect it in production

  • Registry and secret-scanning alerts on new images; docker history --no-trunc checks in CI.
  • Cloud audit logs for access-key use from unexpected IPs or regions; billing anomaly alerts.
  • Periodic review of public repositories and registries.

Key points

  • A file deleted in a later layer is still in the image.
  • Keep secrets out of the build context, mount them with BuildKit secrets, and prefer short-lived credentials.
  • A leaked secret must be rotated; deleting it isn't enough.

Practice questions

Why doesn't `RUN rm .env` remove a secret from a Docker image?

Answer

Each instruction creates a layer. The file stays in the layer that added it; the deletion only hides it in later layers. Anyone with the image can extract earlier layers.

How do you use a secret during a Docker build without storing it in the image?

Answer

BuildKit secret mounts: RUN --mount=type=secret,id=... exposes the secret only to that step, and docker build --secret id=...,src=... provides it. Exclude secrets from the context with .dockerignore.

A secret was pushed in an image to a public registry. What do you do?

Answer

Rotate or revoke it immediately, check audit logs for use, remove the image, then fix the build (secret mounts, .dockerignore) and add secret scanning so it can't recur.

Open in the app to mark lessons complete, take notes and track your progress.

Was this page helpful?