Why split files into content-addressed chunks in a file sync service?

Each chunk is named by its hash, so the client uploads only chunks the server lacks: a small edit re-uploads one chunk, an interrupted upload resumes from where it stopped, and identical chunks (in other files or versions) are stored once.

How do devices stay in sync after being offline for a long time?

Each user has an ordered change journal. A device stores the position (cursor) of the last change it applied and asks for all changes after it, then downloads the chunks it's missing. The same mechanism handles one second or one week offline.

Design Cloud File Storage and Sync — System Design Interviews, Step by Step

The question

"Design a cloud file storage service: users upload files from any device, see them everywhere, and changes sync automatically."

Clarify the requirements

Functional: upload, download, delete, folders; automatic sync between a user's devices; share a file or folder by link; keep previous versions for 30 days.
Scale: 50 million users, 10 million daily active; average 200 files per user; files from 1 KB to 10 GB.
Non-functional: never lose a file (durability first); uploads resume after a dropped connection; changes appear on other devices within seconds; large files shouldn't re-upload entirely after a small edit.

Estimate the scale

Files: 50M × 200 = 10 billion files; at an average of 1 MB that's 10 PB before replication: object storage, not a database.
Metadata: 10B rows × ~500 bytes ≈ 5 TB, a sharded database.
Changes: say 10M daily users × 20 file changes ≈ 200M/day ≈ 2,300 changes/s.

API and data model

POST /files/upload-session  { path, size, chunk_hashes[] } -> { missing_chunks[], upload_urls }
PUT  {upload_url}           (chunk bytes, straight to object storage)
POST /files/commit          { path, chunk_hashes[], base_version } -> { file_id, version }
GET  /changes?cursor=...    -> { changes: [...], next_cursor }   (long-poll or push trigger)
GET  /files/{id}/download   -> chunk list with signed URLs

files:    file_id | owner_id | path | latest_version | deleted
versions: file_id | version | chunk_hashes (ordered) | size | created_at | device_id
chunks:   hash (PK) | size | storage_key | ref_count
journal:  owner_id | seq | file_id | action | version     (one ordered change log per user)

High-level design

device ─► API (metadata service) ─► metadata DB (sharded by owner)
   │            │                         └─► change journal ─► notification service ─► other devices
   │            └─► signed upload URLs
   └──────────────► object storage (chunks, content-addressed by hash)

The client splits a file into chunks (e.g. 4 MB), hashes each, and asks which chunks the server doesn't have yet.
It uploads only the missing chunks directly to object storage with signed URLs (app servers never carry file bytes).
It commits the new version (the ordered list of chunk hashes); the metadata service appends to the user's change journal and notifies the user's other devices, which fetch the changes and download missing chunks.

Deep dives

1. Chunking and deduplication. Content-addressed chunks (named by their hash) give three wins at once: a small edit re-uploads one chunk instead of the whole file, identical chunks are stored once, and an interrupted upload resumes from the chunks already sent.

Only changed chunks are uploaded; identical chunks are stored onceTry it yourself

import hashlib

CHUNK = 8          # bytes, tiny for the demo; ~4 MB in real systems
store = {}         # object storage: hash -> bytes


def chunks(data):
    return [data[i:i + CHUNK] for i in range(0, len(data), CHUNK)]


def upload(data):
    hashes, sent = [], 0
    for piece in chunks(data):
        h = hashlib.sha256(piece).hexdigest()[:10]
        if h not in store:          # the server asks only for chunks it lacks
            store[h] = piece
            sent += len(piece)
        hashes.append(h)
    return hashes, sent


def download(hashes):
    return b"".join(store[h] for h in hashes)


v1 = b"Quarterly report: revenue up. Costs flat. Hiring paused. Outlook good."
v2 = b"Quarterly report: revenue up. Costs down. Hiring paused. Outlook good."
copy = v1                                     # another user uploads the same file

h1, sent1 = upload(v1)
h2, sent2 = upload(v2)
h3, sent3 = upload(copy)
print(f"v1: {len(v1)} bytes, uploaded {sent1}")
print(f"v2 (one word changed): uploaded {sent2}")
print(f"identical copy: uploaded {sent3}")
print("stored chunks:", len(store), "| v2 restores correctly:", download(h2) == v2)

import hashlib

CHUNK = 8          # bytes, tiny for the demo; ~4 MB in real systems
store = {}         # object storage: hash -> bytes

def chunks(data):
    return [data[i:i + CHUNK] for i in range(0, len(data), CHUNK)]

def upload(data):
    hashes, sent = [], 0
    for piece in chunks(data):
        h = hashlib.sha256(piece).hexdigest()[:10]
        if h not in store:          # the server asks only for chunks it lacks
            store[h] = piece
            sent += len(piece)
        hashes.append(h)
    return hashes, sent

def download(hashes):
    return b"".join(store[h] for h in hashes)

v1 = b"Quarterly report: revenue up. Costs flat. Hiring paused. Outlook good."
v2 = b"Quarterly report: revenue up. Costs down. Hiring paused. Outlook good."
copy = v1                                     # another user uploads the same file

h1, sent1 = upload(v1)
h2, sent2 = upload(v2)
h3, sent3 = upload(copy)
print(f"v1: {len(v1)} bytes, uploaded {sent1}")
print(f"v2 (one word changed): uploaded {sent2}")
print(f"identical copy: uploaded {sent3}")
print("stored chunks:", len(store), "| v2 restores correctly:", download(h2) == v2)

Output

v1: 70 bytes, uploaded 70
v2 (one word changed): uploaded 8
identical copy: uploaded 0
stored chunks: 10 | v2 restores correctly: True

Fixed-size chunks are simple, but inserting a byte near the start shifts every later chunk. Real sync clients often use content-defined chunking (boundaries chosen by a rolling hash of the content) so an insertion only changes one or two chunks.

2. Sync with a change journal. Each user has an ordered journal of changes. Every device remembers the last journal position it applied (a cursor) and asks for "changes since my cursor". A long-poll or push message tells devices there's something new; the journal makes catching up after a week offline the same operation as catching up after a second.

3. Conflicts. Two devices edit the same file offline. The commit carries base_version; if the server's latest version is newer, the second commit is a conflict. Don't merge binary files automatically: keep both, saving the second as "report (conflicted copy from laptop).docx", and let the user decide.

4. Durability. Object storage replicates chunks across devices and data centres. Delete a chunk only when no version references it (ref_count reaches zero) and the 30-day version history has expired.

Trade-offs

Chunk size: small chunks dedupe better but mean more metadata and more requests; ~4 MB is a common balance.
Dedup scope: across all users saves the most space but can leak information ("this file already exists"); per-user dedup is safer.
Conflict handling: "conflicted copies" are safe and simple; real-time merging only works for document formats designed for it.
Notifications: long-polling is simple; WebSockets or mobile push save battery and requests.

What interviewers look for

Separating metadata (database) from file bytes (object storage), with direct uploads via signed URLs.
Chunking for resumable uploads, deltas and dedup.
A per-user change journal with cursors for sync.
A clear conflict policy.

Key points

Files → content-addressed chunks in object storage; metadata in a sharded DB.
Upload only missing chunks; resume and dedupe for free.
Devices sync from a per-user journal using a cursor.
Conflicts become "conflicted copies"; never silently overwrite.