Manual Changes That Quietly Undo GitOps

Scenario

Example company: during an incident, an engineer scales the checkout Deployment to 10 replicas with kubectl scale and raises a memory limit with kubectl edit. Argo CD manages the namespace with automated sync but without self-heal.

What you see in production

The cluster shows the manual values for days, Git still says 3 replicas, and Argo CD shows the app as OutOfSync, which the team learns to ignore. A week later, an unrelated commit triggers a sync that silently reverts both changes during peak traffic.

The naive code

syncPolicy:
  automated: {}            # sync when Git changes; manual drift is left alone until then

Reproduce it

A small reconcile loop with and without self-heal, facing manual edits and a later Git commit:

def reconcile(git, live, self_heal, git_changed):
    drift = {k: (git[k], live.get(k)) for k in git if live.get(k) != git[k]}
    if drift and (self_heal or git_changed):
        live.update(git)                          # Argo CD applies the desired state from Git
        return "synced", drift
    return ("OutOfSync" if drift else "Synced"), drift

for self_heal in (False, True):
    git = {"replicas": 3, "memory_limit": "512Mi"}
    live = dict(git)
    log = []
    live.update(replicas=10, memory_limit="1Gi")                       # kubectl scale / kubectl edit
    log.append(reconcile(git, live, self_heal, git_changed=False)[0])  # next reconcile minutes later
    log.append(reconcile(git, live, self_heal, git_changed=False)[0])  # days pass
    git["image"] = "checkout:2.8.1"; live.setdefault("image", "checkout:2.8.0")
    status, reverted = reconcile(git, live, self_heal, git_changed=True)
    log.append(f"commit -> {status}, reverted {sorted(k for k in reverted if k != 'image')}")
    print(f"selfHeal={self_heal!s:5}: " + " | ".join(log))

selfHeal=False: OutOfSync | OutOfSync | commit -> synced, reverted ['memory_limit', 'replicas']
selfHeal=True : synced | Synced | commit -> synced, reverted []

Root cause

With automated sync but no self-heal, Argo CD only applies Git when Git changes; manual changes persist as drift until any later sync reverts them at an unpredictable moment. The deeper problem is having two sources of truth: the cluster and Git.

The fix

syncPolicy:
  automated:
    prune: true          # delete resources removed from Git
    selfHeal: true       # revert manual changes to the live state
  syncOptions: [CreateNamespace=true]

• Make Git the only way to change managed resources; restrict kubectl write access in production with RBAC.
• For emergencies, change Git (a fast-track PR) or temporarily disable auto-sync for that app, with an expiry and a follow-up PR.
• Exclude fields legitimately owned by controllers with ignoreDifferences (for example spec.replicas when an HPA manages it).
• Alert on apps OutOfSync for more than a few minutes.

Trade-offs

• Self-heal reverts emergency fixes within seconds unless the process accounts for it.
• prune: true deletes resources removed from Git; protect important ones with Prune=false annotations and review deletions.

Detect it in production

• Argo CD application health and sync status; alerts on long OutOfSync.
• Kubernetes audit logs of human writes to GitOps-managed namespaces.

Key points

• Without self-heal, manual changes linger until a random later sync reverts them.
• Enable selfHeal and prune, and make Git the only path to production changes.
• Use ignoreDifferences for fields other controllers own, such as HPA-managed replicas.

Further reading: Argo CD automated sync policy · Diffing customization