Cloud Computing Tutorial · Storage and Databases · Lesson 15 of 24
Object Storage (S3, Blob, Cloud Storage)
Store files in buckets: keys, storage classes, lifecycle rules, versioning, presigned URLs and keeping buckets private.
Buckets and objects
Object storage (Amazon S3, Azure Blob Storage, Google Cloud Storage) stores files ("objects") under keys in buckets, with practically unlimited capacity and very high durability. It's used for uploads, backups, logs, data lakes, static websites and ML datasets.
Storage classes and lifecycle rules
Pay less for data you rarely read:
| Access pattern | AWS | Azure | Google Cloud |
|---|---|---|---|
| Frequent | S3 Standard | Hot | Standard |
| Infrequent (monthly) | S3 Standard-IA | Cool | Nearline |
| Rare (quarterly) | Glacier Instant Retrieval | Cold | Coldline |
| Archive (yearly, hours to restore) | Glacier Deep Archive | Archive | Archive |
| Unknown | S3 Intelligent-Tiering | (lifecycle management) | Autoclass |
Lifecycle rules move or delete objects by age automatically.
PRICE = {"Standard": 0.023, "Standard-IA": 0.0125, "Glacier Instant": 0.004, "Deep Archive": 0.00099} # example $/GB-month
logs_gb_per_month, months = 2_000, 24
def cost(rule):
total = 0
for age in range(months): # each month's logs, aged 0..23 months
cls = rule(age)
total += logs_gb_per_month * PRICE[cls]
return total
print(f"keep everything in Standard: ${cost(lambda age: 'Standard'):8,.0f} / month after 2 years")
tiered = lambda age: "Standard" if age < 1 else "Standard-IA" if age < 3 else "Glacier Instant" if age < 12 else "Deep Archive"
print(f"lifecycle rule (1/3/12 months): ${cost(tiered):6,.0f} / month")
Output
keep everything in Standard: $ 1,104 / month after 2 years lifecycle rule (1/3/12 months): $ 192 / month
Safety
- Keep buckets private (S3 Block Public Access, Azure "allow blob public access" off, Google public access prevention); serve public files through a CDN.
- Turn on versioning for important data, and object lock (WORM) for backups that ransomware must not delete.
- Share single files with presigned URLs (SAS tokens on Azure, signed URLs on Google Cloud) that expire.
import hashlib, hmac
from urllib.parse import urlencode
SECRET = b"server-side-signing-key" # real presigned URLs use your credentials (SigV4)
def presign(path, expires_at):
sig = hmac.new(SECRET, f"GET\n{path}\n{expires_at}".encode(), hashlib.sha256).hexdigest()[:16]
return f"https://files.example.com{path}?" + urlencode({"expires": expires_at, "sig": sig})
def verify(path, expires_at, sig, now):
good = hmac.new(SECRET, f"GET\n{path}\n{expires_at}".encode(), hashlib.sha256).hexdigest()[:16]
return hmac.compare_digest(good, sig) and now < expires_at
url = presign("/invoices/42.pdf", expires_at=1_000_900)
sig = url.rsplit("sig=", 1)[1]
print(url)
print("valid now: ", verify("/invoices/42.pdf", 1_000_900, sig, now=1_000_000))
print("valid after expiry: ", verify("/invoices/42.pdf", 1_000_900, sig, now=1_001_000))
print("other file, same sig:", verify("/invoices/43.pdf", 1_000_900, sig, now=1_000_000))
Output
https://files.example.com/invoices/42.pdf?expires=1000900&sig=74872abb28da9822 valid now: True valid after expiry: False other file, same sig: False
Test yourself
How should users download private invoices stored in S3?
Make the bucket publicThrough short-lived presigned URLs generated by the appEmail them the access keysCopy files to every web server
Show answer
Answer: Through short-lived presigned URLs generated by the app
Presigned URLs grant access to one object for a limited time without opening the bucket.