Cloud Engineering in Production · Security Incidents
The Key on GitHub
Long-lived access keys leak into repositories, logs and laptops. Remove them: SSO for people, OIDC for CI, roles for workloads, and guardrails for the rest.
Scenario
Example team: a developer debugging a deployment script pastes an IAM user's access key into a config file and pushes it to a public GitHub repository at 23:10. The IAM user belongs to a "deployers" group with AdministratorAccess, created three years ago for a CI server.
What you see in production
By 23:18, hundreds of large GPU instances are launching in regions the company has never used. By morning the bill estimate is over $40,000, AWS has opened a support case about compromised credentials, and nobody is sure what else the attacker created (new IAM users? access to S3 data?).
The naive code
# deploy/config.ini, committed to a public repository
[aws]
aws_access_key_id = AKIA................
aws_secret_access_key = ........................................
with a long-lived IAM user, admin permissions, no region guardrails and no budget alerts.
Reproduce it
What an attacker can do with a leaked key depends entirely on the permissions behind it and the guardrails around it:
ACTIONS = ["ec2:RunInstances (crypto mining)", "iam:CreateUser (persistence)", "s3:GetObject (customer data)",
"kms:Decrypt", "organizations:LeaveOrganization"]
def allowed(action, setup):
service = action.split(":")[0]
if setup["scp_region_lock"] and service == "ec2" and setup["attacker_region"] not in setup["allowed_regions"]:
return False
if setup["scp_deny_leave_org"] and action.startswith("organizations:LeaveOrganization"):
return False
return service in setup["permissions"]
setups = {
"admin IAM user, no guardrails": {"permissions": {"ec2", "iam", "s3", "kms", "organizations"},
"scp_region_lock": False, "scp_deny_leave_org": False},
"least-privilege deploy role + SCPs": {"permissions": {"s3"}, "scp_region_lock": True, "scp_deny_leave_org": True},
}
for name, setup in setups.items():
setup.update(attacker_region="ap-southeast-3", allowed_regions={"eu-west-1"})
possible = [a for a in ACTIONS if allowed(a, setup)]
print(f"{name}: attacker can do {len(possible)}/{len(ACTIONS)}")
for a in possible:
print(" -", a)
Output
admin IAM user, no guardrails: attacker can do 5/5 - ec2:RunInstances (crypto mining) - iam:CreateUser (persistence) - s3:GetObject (customer data) - kms:Decrypt - organizations:LeaveOrganization least-privilege deploy role + SCPs: attacker can do 1/5 - s3:GetObject (customer data)
Root cause
A long-lived credential existed at all, it had far more permission than the job needed, and there were no guardrails (region restrictions, budget alerts, threat detection) to limit or reveal misuse. Secrets in Git are found by automated scanners within minutes, and deleting the commit doesn't help once it has been cloned.
The fix
Respond now: deactivate and delete the key; check CloudTrail for everything it did (new users, roles, keys, instances, data access) and remove persistence; terminate the instances; contact provider support; rotate any other secrets in that repository.
Remove long-lived keys:
| Who | AWS | Azure | Google Cloud |
|---|---|---|---|
| People | IAM Identity Center (SSO) + MFA | Entra ID + MFA, Privileged Identity Management | Cloud Identity / Workspace + 2SV |
| CI/CD | OIDC federation to an IAM role | Workload identity federation | Workload Identity Federation |
| Workloads | IAM roles (instance profiles, IRSA/Pod Identity, Lambda roles) | Managed identities | Attached service accounts; disable service-account key creation by org policy |
See GitHub Actions OIDC in AWS and Google's guidance on service account keys.
Add guardrails: service control policies that deny unused regions and deny disabling CloudTrail or GuardDuty; budgets and cost anomaly detection; GuardDuty for crypto-mining and credential-misuse findings; secret scanning with push protection on every repository.
Trade-offs
- SSO and federation need setup and a break-glass path, but remove the whole class of leaked-key incidents.
- Region-deny SCPs must exempt global services (IAM, CloudFront, Route 53) and need updating when you expand.
- Push protection occasionally blocks test fixtures; allow-list them explicitly rather than turning it off.
Detect it in production
- GuardDuty (Defender for Cloud, Security Command Center) findings for unusual API calls and mining.
- CloudTrail alerts on
CreateUser,CreateAccessKey, and API calls from new regions or IPs. - Cost anomaly alerts; a report of any remaining IAM users with access keys (should be zero).
Key points
- Long-lived keys will leak eventually; replace them with SSO, OIDC federation and roles.
- Limit what any credential can do: least privilege plus organisation guardrails.
- Detect misuse early with threat detection, CloudTrail alerts and cost anomalies.
Practice questions
An AWS access key was pushed to a public repository. What do you do, in order?
Answer
Deactivate and delete the key, investigate its use in CloudTrail (instances, IAM users, roles, keys, data access), remove anything the attacker created, contact support, rotate other exposed secrets, then fix the root cause by moving to federation and roles.
How should GitHub Actions authenticate to AWS without stored keys?
Answer
With OIDC: GitHub issues a short-lived token per workflow run, and an IAM role trusts that token for a specific repository and branch, returning temporary credentials.
What are the Azure and Google Cloud equivalents for removing stored credentials?
Answer
Azure: managed identities for workloads and workload identity federation for CI. Google Cloud: attached service accounts, Workload Identity Federation, and an organisation policy that disables service-account key creation.
Open in the app to mark lessons complete, take notes and track your progress.