Sign in Get started

Cloud Engineering in Production · Capstone Case Studies

Capstone: Postmortem — The Weekend That Cost $52,000

A leaked key, a public bucket and an alert nobody received. Reconstruct the incident and write the remediation plan.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

Example company: a SaaS analytics company. On Saturday at 02:14, GPU instances started launching in four regions the company never uses. Cost alerts existed but went to a finance mailbox. On Monday at 08:30 an engineer noticed the console looked strange. Total unexpected spend: $52,000. The investigation also found that the attacker downloaded a CSV export of customer contacts.

Evidence

  • The attacker used the access key of IAM user ci-legacy, created in 2021 for Jenkins, with AdministratorAccess.
  • The key had been printed in a CI log (debug mode) that was attached to a public GitHub issue on Friday at 18:02.
  • CloudTrail shows RunInstances in 4 regions, CreateUser (support-admin), CreateAccessKey, and ListBuckets/GetObject on acme-exports.
  • The acme-exports bucket held customer CSVs from the sales team; the attacker could read it because the key was an admin, not because the bucket was public.
  • GuardDuty was enabled only in the primary region; budgets alerted by email to finance@.
  • No SCP restricted regions.

Deliverables

  1. Timeline and root cause.
  2. Contributing factors.
  3. Immediate containment and recovery steps (in order).
  4. Structural fixes and how you'll verify them.

Draft each deliverable before revealing the model answer and rubric.

Further reading: Remediating compromised AWS credentials · GuardDuty

Deliverables

Timeline and root cause.

Model answer

Fri 18:02 key exposed in a public issue → Sat 02:14 first RunInstances in unused regions → CreateUser/CreateAccessKey for persistence → reads of acme-exports → Mon 08:30 detection (54 hours later).

Root cause: a long-lived, admin-privileged access key existed for a legacy CI user and was printed in logs that became public.

Rubric

Full marks: clear timeline including persistence and data access, root cause framed as the existence and privilege of the long-lived key.

Contributing factors.

Model answer

  • Admin permissions on a CI identity; no least privilege.
  • Debug logging that printed credentials; CI logs shared publicly.
  • No region-deny SCP; GuardDuty only in one region.
  • Cost alerts routed to a finance mailbox, not on-call; no anomaly alert paging.
  • Customer data in a general-purpose bucket readable by any admin principal; no Macie or data-access alerts.

Rubric

Full marks: at least four factors across identity, detection, guardrails and data governance; blameless framing.

Immediate containment and recovery, in order.

Model answer

  1. Deactivate ci-legacy's keys and the attacker's support-admin user and keys; attach a deny-all policy to both.
  2. Search CloudTrail in all regions for everything created by those principals (instances, roles, users, keys, Lambda functions, snapshots shares) and remove it; terminate instances in all regions.
  3. Open a support case for compromised credentials and cost review.
  4. Rotate all secrets the CI system held; check for other leaked credentials in logs.
  5. Assess the data exposure with legal/privacy: which customers, notification obligations.

Rubric

Full marks: revokes all attacker access including persistence before cleanup, checks all regions, involves support and legal.

Structural fixes and verification.

Model answer

Fixes: delete all IAM users with keys (CI via OIDC roles, people via SSO); SCPs denying unused regions and IAM user/key creation; GuardDuty and Security Hub in all regions via delegated admin; cost anomaly and budget alerts to the on-call pager; CI log redaction and secret scanning with push protection; customer exports in a dedicated bucket with least-privilege access, Macie and data-event logging.

Verify: a report showing zero IAM access keys; an attempted RunInstances in a denied region fails; a GuardDuty sample finding pages on-call; a simulated cost anomaly reaches the pager; quarterly review of these checks.

Rubric

Full marks: eliminates long-lived keys, adds preventive and detective controls, fixes alert routing, and proves each control with a test.

Open in the app to mark lessons complete, take notes and track your progress.

Was this page helpful?