Promotion Between Environments by Copy and Paste

Scenario

Example company: to release, an engineer edits the image tag in overlays/prod to "the version that's in staging". One Friday they copy the tag from the wrong service's overlay.

What you see in production

Production runs a build that never ran in staging. Rolling back is easy, but nobody can answer "which version passed which tests" without digging through CI logs.

The naive code

sed -i 's/newTag: .*/newTag: 2.9.0/' overlays/prod/kustomization.yaml   # by hand, from memory
git commit -am "prod release" && git push

Reproduce it

A promotion pipeline that only moves versions which passed the previous stage's checks:

stages = ["dev", "staging", "prod"]
deployed = {"dev": "2.9.1", "staging": "2.9.0", "prod": "2.8.4"}
checks = {("2.9.0", "staging"): {"smoke": True, "e2e": True, "soak_2h": True},
          ("2.9.1", "dev"): {"smoke": True, "e2e": False}}

def promote(version, frm, to):
    if stages.index(to) != stages.index(frm) + 1:
        return f"refused: {frm} -> {to} skips a stage"
    if deployed[frm] != version:
        return f"refused: {version} is not what runs in {frm} ({deployed[frm]})"
    results = checks.get((version, frm), {})
    if not results or not all(results.values()):
        failed = [k for k, ok in results.items() if not ok] or ["no results"]
        return f"refused: {version} failed {failed} in {frm}"
    deployed[to] = version
    return f"promoted {version}: {frm} -> {to} (pull request opened with the new tag)"

print(promote("2.9.1", "staging", "prod"))      # the copy-paste mistake
print(promote("2.9.1", "dev", "staging"))
print(promote("2.9.0", "staging", "prod"))

refused: 2.9.1 is not what runs in staging (2.9.0)
refused: 2.9.1 failed ['e2e'] in dev
promoted 2.9.0: staging -> prod (pull request opened with the new tag)

Root cause

When promotion is a manual edit, nothing guarantees that the version reaching production is the one that passed earlier stages. Tags typed by hand can come from the wrong place, and the record of what was tested lives in people's heads.

The fix

• Promote the same artefact (image digest) from stage to stage; never rebuild per environment.
• Automate promotion: a pipeline or tool (Kargo, Argo CD Image Updater with write-back, a CI job) opens a pull request bumping the next environment's tag only when the previous stage's checks passed.
• Make checks explicit and recorded: smoke tests, end-to-end tests, soak time, approvals for production.
• Keep promotion in Git so every change is reviewable and revertible.

Trade-offs

• Automated promotion needs reliable tests; flaky tests block releases.
• Extra tooling (Kargo, image updaters) is another system to learn and operate.

Detect it in production

• Versions running in production that never ran in staging (compare deployment history).
• Promotion PRs without linked test results.

Key points

• Manual tag edits let untested builds reach production.
• Promote the same digest through stages, gated on recorded checks.
• Do it through pull requests so promotion is auditable and reversible.

Further reading: Argo CD Image Updater · Kargo