Helm, Kustomize and GitOps with Argo CD · GitOps with Argo CD
Secrets Committed to the GitOps Repository
A Secret manifest in Git is plaintext for anyone with repository access; encrypt it or keep only references in Git.
Scenario
Example company: to make the GitOps repository "complete", a team commits Kubernetes Secret manifests for database passwords and API keys. The repository is shared with 60 engineers and a contractor.
What you see in production
A secret scanner flags the repository. Everyone with read access, every CI runner that clones it and every fork has had the production database password for months. The values are only base64-encoded.
The naive code
apiVersion: v1
kind: Secret
metadata: {name: db}
data:
password: cHJvZC1QQHNzdzByZA== # base64, not encryption
Reproduce it
Decode the committed value, then compare with an encrypted value that only the cluster's key can open (the idea behind Sealed Secrets and SOPS):
echo "committed value decodes to: $(echo 'cHJvZC1QQHNzdzByZA==' | base64 -d)"
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out cluster-key.pem 2> /dev/null # stays in the cluster
openssl pkey -in cluster-key.pem -pubout -out cluster-pub.pem # can be shared
printf 'prod-P@ssw0rd' | openssl pkeyutl -encrypt -pubin -inkey cluster-pub.pem | base64 -w0 > sealed.txt
echo "what Git stores instead: $(cut -c1-48 sealed.txt)... ($(wc -c < sealed.txt) characters)"
echo "decrypted inside the cluster: $(base64 -d sealed.txt | openssl pkeyutl -decrypt -inkey cluster-key.pem)"
Output
committed value decodes to: prod-P@ssw0rd what Git stores instead: XgTtOO2WSZOzdXTbTFpsP48umvvTT8IH4HIfqDnQaldW3cd1... (344 characters) decrypted inside the cluster: prod-P@ssw0rd
Run this on your computer — see “Get Started” for setup.
Root cause
Kubernetes Secrets are base64-encoded, not encrypted. Committing them to Git publishes the values to everyone and everything with repository access, and Git history keeps them forever, even after deletion.
The fix
Choose one pattern and enforce it:
- External Secrets Operator: Git contains an
ExternalSecretthat references a key in a secret manager (AWS Secrets Manager, Vault, GCP Secret Manager); the operator creates the Secret in the cluster. - Sealed Secrets: Git contains values encrypted with the cluster's public key; only the controller in the cluster can decrypt.
- SOPS (with age or cloud KMS): encrypted values in Git, decrypted at deploy time by Argo CD plugins or CI.
Plus: secret scanning on every push, and rotate any secret that ever reached Git.
Trade-offs
- External secret managers add a runtime dependency but give rotation and audit.
- Sealed Secrets are per cluster; re-encrypt for each cluster and back up the controller key.
- SOPS keeps everything in Git but needs key distribution to deploy tooling.
Detect it in production
- Secret scanners (gitleaks, GitHub secret scanning) in CI and on the whole history.
- Policy that rejects
kind: Secretin GitOps repositories.
Key points
- Base64 Secrets in Git are plaintext to every reader, forever.
- Store references (External Secrets) or encrypted values (Sealed Secrets, SOPS) instead.
- Scan repositories and rotate anything that leaked.
Further reading: Argo CD secret management · External Secrets Operator · Sealed Secrets
Practice questions
Why is committing a Kubernetes Secret manifest to Git a leak?
Answer
Secret data is only base64-encoded, so anyone with repository access can decode it, and Git history keeps it even after deletion.
How does the External Secrets Operator keep secrets out of Git?
Answer
Git holds an ExternalSecret that references a key in an external secret manager; the operator fetches the value and creates the Kubernetes Secret in the cluster.
What must happen after a secret is found in Git history?
Answer
Rotate it immediately (the old value is compromised), then remove it from the repository and prevent recurrence with scanning and policy.
Open in the app to mark lessons complete, take notes and track your progress.