What's the difference between RPO and RTO?

RPO is the maximum acceptable data loss measured in time (how old the last recoverable point may be); RTO is the maximum acceptable downtime before service is restored.

How do you protect backups from ransomware or a malicious administrator?

Copy them to a separate account (and region) with restricted access and make them immutable, e.g. AWS Backup Vault Lock in compliance mode, Azure immutable vaults, or locked retention on Google Cloud backup vaults.

How do you know your backups work?

Restore them regularly: automated restore tests that validate the data and measure restore time, plus periodic full recovery drills with the runbook.

The Backup That Didn't Restore — Cloud Engineering in Production

Scenario

Example team: an engineer runs a migration script against production instead of staging and deletes 30% of the orders table. "No problem, we have backups." RDS automated backups are on with 7-day retention, and a nightly pg_dump job writes to S3.

What you see in production

The point-in-time restore creates a new database instance, which takes about an hour; the application config has to be changed to point at it; then the team discovers the nightly pg_dump had been failing silently for three weeks (disk full on the job server), and that the restored instance is missing a parameter group and two read replicas. Total recovery: nine hours, plus manual reconciliation of orders placed after the restore point.

The naive code

RDS automated backups: 7 days     (never restored as a drill)
nightly pg_dump -> S3              (failures only visible in a log file nobody reads)
same account for backups and production; anyone with admin can delete them

Reproduce it

Compare the strategy's real recovery time and data loss with what the business needs:

RPO and RTO of the backup strategyTry it yourself

incident_minute = 14 * 60 + 37           # bad script ran at 14:37
strategies = {
    "nightly pg_dump (failing for 3 weeks)": {"last_good": incident_minute - 21 * 24 * 60, "restore_min": 240},
    "RDS point-in-time restore": {"last_good": incident_minute - 1, "restore_min": 60 + 45},   # restore + reconfigure
    "PITR + runbook + IaC for config": {"last_good": incident_minute - 1, "restore_min": 35},
}
RPO_TARGET, RTO_TARGET = 5, 60

for name, s in strategies.items():
    rpo = incident_minute - s["last_good"]
    rto = s["restore_min"]
    ok = "meets" if rpo <= RPO_TARGET and rto <= RTO_TARGET else "MISSES"
    print(f"{name:38} data lost {rpo:6} min, down {rto:4} min -> {ok} RPO {RPO_TARGET} / RTO {RTO_TARGET}")

Output

nightly pg_dump (failing for 3 weeks)  data lost  30240 min, down  240 min -> MISSES RPO 5 / RTO 60
RDS point-in-time restore              data lost      1 min, down  105 min -> MISSES RPO 5 / RTO 60
PITR + runbook + IaC for config        data lost      1 min, down   35 min -> meets RPO 5 / RTO 60

Root cause

Backups were configured but never restored under realistic conditions, so nobody knew the real RTO, the runbook didn't exist, and the failing dump job went unnoticed. Backups lived in the same account with the same admin permissions as production, so a mistake or an attacker could have deleted them too.

The fix

Set RPO and RTO per system with the business, and choose mechanisms to match (point-in-time recovery for minutes of RPO, replicas or standby for minutes of RTO).
Centralise backups with AWS Backup (Azure Backup, Google Cloud Backup and DR) using policies applied by tag, so new databases are covered automatically.
Copy backups to another account and region, and make them immutable with Vault Lock (compliance mode) or the equivalents, so neither mistakes nor ransomware can delete them.
Test restores on a schedule (AWS Backup restore testing can do it automatically) and run a full recovery drill with the runbook at least quarterly; measure the actual RTO.
Monitor backup jobs: alert on failed or missing backups, not just on errors in logs.
Make recovery fast: infrastructure and configuration as code so a restored database gets the right parameters, replicas and connection settings in minutes.

Trade-offs

Cross-account, cross-region, immutable copies cost storage and transfer; size retention by business need.
Compliance-mode vault lock can't be undone, even by administrators; test with governance mode first.
Restore drills take engineering time, which is how you find out the RTO before an incident does.

Detect it in production

Backup coverage report: every production data store has a backup plan (by tag).
Alerts on failed backup and copy jobs; age of the newest restorable recovery point per resource.
Restore test results and measured RTO per system, trended.

Key points

A backup that hasn't been restored is a hope, not a plan.
Define RPO/RTO, centralise backups, copy them to another account and region, and make them immutable.
Automate restore tests and monitor backup jobs.