Same Commit, Different Build

Scenario

Example team: an urgent one-line fix needs to go out on Saturday. The pipeline rebuilds the image from the same Dockerfile as Thursday's release: FROM python:3-slim, pip install -r requirements.txt with some unpinned packages, and the Kubernetes Deployment references api:latest.

What you see in production

The hot-fix image fails at startup with an import error from a library nobody changed. Half the pods run Thursday's latest, half Saturday's (pods that restarted pulled the new latest). Rolling back "to the previous image" is impossible: nobody knows which digest Thursday's latest was.

The naive code

FROM python:3-slim                      # floats across minor Python versions
RUN pip install -r requirements.txt     # "flask>=2" style ranges

containers:
  - name: api
    image: registry.example.com/api:latest
    imagePullPolicy: Always

Reproduce it

A tag is a movable pointer; a digest is the content's hash and never changes:

import hashlib

def digest(content):
    return "sha256:" + hashlib.sha256(content.encode()).hexdigest()[:12]

registry = {}
def push(tag, content):
    registry[tag] = digest(content)
    return registry[tag]

thursday = push("python:3-slim", "python 3.13.7 + openssl 3.5.1")
saturday = push("python:3-slim", "python 3.14.0 + openssl 3.5.2")   # upstream published a new image
print("Thursday python:3-slim ->", thursday)
print("Saturday python:3-slim ->", saturday, "(same tag, different content)")

pinned = f"python:3.13-slim@{thursday}"
print("pinned reference always resolves to:", pinned.split("@")[1])

Thursday python:3-slim -> sha256:32a52ed0df42
Saturday python:3-slim -> sha256:ef21bde81603 (same tag, different content)
pinned reference always resolves to: sha256:32a52ed0df42

Root cause

Tags like latest, 3-slim or 22 are re-pointed by publishers whenever they release, and unpinned dependency ranges resolve to whatever is newest at build time. Rebuilding "the same commit" therefore produces a different artifact. Deploying a floating tag makes it worse: what runs depends on when each node last pulled, and the previous version can't be identified for rollback.

The fix

1. Pin base images to a version and digest; let a bot propose updates:

FROM python:3.13-slim@sha256:<digest>

2. Lock dependencies (requirements.txt with hashes via pip-tools or uv, package-lock.json, go.sum) and install from the lock file (npm ci, pip install --require-hashes).
3. Build once, deploy many: tag images with the commit SHA, push once, promote that exact image through environments, and deploy by digest. Never deploy latest.
4. Keep releases rebuildable: retain images for released versions; record the digest in the deployment history so rollback is "redeploy digest X".
5. Update on purpose: Renovate or Dependabot opens PRs for new base images and packages, and CI tests them like any change.

image: registry.example.com/api@sha256:<digest>   # or api:3f9c2a1 with immutable tags enforced in the registry

Trade-offs

• Pinned digests go stale and miss security patches unless an update bot and regular merges keep them fresh.
• Immutable tags and retention policies need registry configuration and storage.
• Lock files need discipline: one tool, regenerated through PRs, not by hand.

Detect it in production

• Admission policy (Kyverno, OPA Gatekeeper) rejecting :latest and requiring digests or immutable SHA tags.
• Report running image digests per environment; alert when pods of one Deployment run different digests.
• Reproducibility check: rebuild a release in CI and compare dependencies.

Key points

• Tags float; digests don't. Floating references make builds and deploys non-reproducible.
• Pin base images and dependencies, build once, deploy by SHA or digest.
• Let bots propose updates so pinning doesn't mean falling behind.