Cloud Engineering in Production · Security Incidents

The Key on GitHub

Long-lived access keys leak into repositories, logs and laptops. Remove them: SSO for people, OIDC for CI, roles for workloads, and guardrails for the rest.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

Example team: a developer debugging a deployment script pastes an IAM user's access key into a config file and pushes it to a public GitHub repository at 23:10. The IAM user belongs to a "deployers" group with AdministratorAccess, created three years ago for a CI server.

What you see in production

By 23:18, hundreds of large GPU instances are launching in regions the company has never used. By morning the bill estimate is over $40,000, AWS has opened a support case about compromised credentials, and nobody is sure what else the attacker created (new IAM users? access to S3 data?).

The naive code

# deploy/config.ini, committed to a public repository
[aws]
aws_access_key_id = AKIA................
aws_secret_access_key = ........................................

with a long-lived IAM user, admin permissions, no region guardrails and no budget alerts.

Reproduce it

What an attacker can do with a leaked key depends entirely on the permissions behind it and the guardrails around it:

Blast radius of one leaked keyTry it yourself
ACTIONS = ["ec2:RunInstances (crypto mining)", "iam:CreateUser (persistence)", "s3:GetObject (customer data)",
           "kms:Decrypt", "organizations:LeaveOrganization"]

def allowed(action, setup):
    service = action.split(":")[0]
    if setup["scp_region_lock"] and service == "ec2" and setup["attacker_region"] not in setup["allowed_regions"]:
        return False
    if setup["scp_deny_leave_org"] and action.startswith("organizations:LeaveOrganization"):
        return False
    return service in setup["permissions"]

setups = {
    "admin IAM user, no guardrails": {"permissions": {"ec2", "iam", "s3", "kms", "organizations"},
                                      "scp_region_lock": False, "scp_deny_leave_org": False},
    "least-privilege deploy role + SCPs": {"permissions": {"s3"}, "scp_region_lock": True, "scp_deny_leave_org": True},
}
for name, setup in setups.items():
    setup.update(attacker_region="ap-southeast-3", allowed_regions={"eu-west-1"})
    possible = [a for a in ACTIONS if allowed(a, setup)]
    print(f"{name}: attacker can do {len(possible)}/{len(ACTIONS)}")
    for a in possible:
        print("   -", a)

Output

admin IAM user, no guardrails: attacker can do 5/5
   - ec2:RunInstances (crypto mining)
   - iam:CreateUser (persistence)
   - s3:GetObject (customer data)
   - kms:Decrypt
   - organizations:LeaveOrganization
least-privilege deploy role + SCPs: attacker can do 1/5
   - s3:GetObject (customer data)

Root cause

A long-lived credential existed at all, it had far more permission than the job needed, and there were no guardrails (region restrictions, budget alerts, threat detection) to limit or reveal misuse. Secrets in Git are found by automated scanners within minutes, and deleting the commit doesn't help once it has been cloned.

The fix

Respond now: deactivate and delete the key; check CloudTrail for everything it did (new users, roles, keys, instances, data access) and remove persistence; terminate the instances; contact provider support; rotate any other secrets in that repository.

Remove long-lived keys:

Who AWS Azure Google Cloud
People IAM Identity Center (SSO) + MFA Entra ID + MFA, Privileged Identity Management Cloud Identity / Workspace + 2SV
CI/CD OIDC federation to an IAM role Workload identity federation Workload Identity Federation
Workloads IAM roles (instance profiles, IRSA/Pod Identity, Lambda roles) Managed identities Attached service accounts; disable service-account key creation by org policy

See GitHub Actions OIDC in AWS and Google's guidance on service account keys.

Add guardrails: service control policies that deny unused regions and deny disabling CloudTrail or GuardDuty; budgets and cost anomaly detection; GuardDuty for crypto-mining and credential-misuse findings; secret scanning with push protection on every repository.

Trade-offs

  • SSO and federation need setup and a break-glass path, but remove the whole class of leaked-key incidents.
  • Region-deny SCPs must exempt global services (IAM, CloudFront, Route 53) and need updating when you expand.
  • Push protection occasionally blocks test fixtures; allow-list them explicitly rather than turning it off.

Detect it in production

  • GuardDuty (Defender for Cloud, Security Command Center) findings for unusual API calls and mining.
  • CloudTrail alerts on CreateUser, CreateAccessKey, and API calls from new regions or IPs.
  • Cost anomaly alerts; a report of any remaining IAM users with access keys (should be zero).

Key points

  • Long-lived keys will leak eventually; replace them with SSO, OIDC federation and roles.
  • Limit what any credential can do: least privilege plus organisation guardrails.
  • Detect misuse early with threat detection, CloudTrail alerts and cost anomalies.

Practice questions

An AWS access key was pushed to a public repository. What do you do, in order?

Answer

Deactivate and delete the key, investigate its use in CloudTrail (instances, IAM users, roles, keys, data access), remove anything the attacker created, contact support, rotate other exposed secrets, then fix the root cause by moving to federation and roles.

How should GitHub Actions authenticate to AWS without stored keys?

Answer

With OIDC: GitHub issues a short-lived token per workflow run, and an IAM role trusts that token for a specific repository and branch, returning temporary credentials.

What are the Azure and Google Cloud equivalents for removing stored credentials?

Answer

Azure: managed identities for workloads and workload identity federation for CI. Google Cloud: attached service accounts, Workload Identity Federation, and an organisation policy that disables service-account key creation.

Open in the app to mark lessons complete, take notes and track your progress.

Was this page helpful?