How do you keep staging and production from drifting apart?

Promote the same artifact, define both from the same IaC and manifests with small explicit overlays, manage dependency versions identically, and detect drift automatically (diffs, GitOps).

Which differences between environments are acceptable?

Explicit, reviewed ones such as scale, endpoints, credentials and data. Versions, configuration keys and feature-flag behaviour should match unless a difference is intentional and documented.

What role does GitOps play against drift?

A controller continuously reconciles the cluster to the Git state, reverting manual changes and making every change reviewable.

It Worked in Staging — DevOps in Production

Scenario

Example team: a release passes every test in staging and fails immediately in production: payments time out. Staging was created by hand two years ago and has been patched independently since.

What you see in production

The investigation finds five differences: production sets HTTP_TIMEOUT=2 while staging uses the default 30; staging runs PostgreSQL 16, production 15; a feature flag is on in staging only; production has a network policy staging lacks; and staging has 2% of production's data. Any one of them could explain "works in staging".

The naive code

staging/   values edited in the console and in a wiki page
prod/      values in a different repo, copied by hand
images     rebuilt per environment from the same branch

Reproduce it

Comparing environments as data makes drift visible, and separates intended differences (sizes, URLs) from accidental ones:

Diff two environmentsTry it yourself

staging = {"image": "api:3f9c2a1", "HTTP_TIMEOUT": "30", "postgres": "16", "flag.new_checkout": "on",
           "replicas": 2, "network_policy": "none", "db_url": "staging-db"}
prod = {"image": "api:3f9c2a1", "HTTP_TIMEOUT": "2", "postgres": "15", "flag.new_checkout": "off",
        "replicas": 12, "network_policy": "deny-egress", "db_url": "prod-db"}
INTENDED = {"replicas", "db_url"}            # differences we expect and document

for key in sorted(set(staging) | set(prod)):
    s, p = staging.get(key), prod.get(key)
    if s != p:
        kind = "intended" if key in INTENDED else "DRIFT"
        print(f"{kind:8} {key:18} staging={s!s:10} prod={p}")

Output

DRIFT    HTTP_TIMEOUT       staging=30         prod=2
intended db_url             staging=staging-db prod=prod-db
DRIFT    flag.new_checkout  staging=on         prod=off
DRIFT    network_policy     staging=none       prod=deny-egress
DRIFT    postgres           staging=16         prod=15
intended replicas           staging=2          prod=12

Root cause

Environments were built and changed by hand, through different paths, so they diverged a little at a time. Only some differences are intentional (scale, endpoints, credentials); the rest are accidents nobody can see. Tests in a non-representative environment give false confidence.

The fix

One artifact, promoted: build the image once and deploy the same digest to every environment.
Configuration as code: a shared base plus small per-environment overlays (Kustomize overlays, Helm values files, Terraform variables), so the only differences are explicit and reviewable.
Same versions of dependencies: databases, brokers and runtimes managed by the same IaC modules in every environment.
Feature flags managed centrally with an audit log, and flag state included in release checks.
Detect drift automatically: diff rendered manifests and Terraform plans between environments in CI; GitOps controllers (Argo CD, Flux) revert manual changes.

deploy/
  base/                 deployment.yaml, service.yaml, config defaults
  overlays/staging/     replicas: 2, db: staging-db
  overlays/prod/        replicas: 12, db: prod-db

Trade-offs

Full production parity (data volume, traffic) is expensive; prioritise parity of versions and configuration, and test scale separately.
Overlays can grow into a second copy of everything; keep them small and review them.
GitOps auto-revert can fight emergency changes; have a documented break-glass path.

Detect it in production

A scheduled job that renders and diffs environments, reporting unexpected differences.
Release checklist: config and flag differences between staging and production shown in the PR.
Track incidents caused by environment differences; they should trend to zero.

Key points

Drift comes from hand-made changes along different paths; it hides bugs until release.
Promote one artifact and keep config as code with explicit, minimal per-environment differences.
Diff environments automatically and let GitOps revert manual changes.