The Demo That Couldn't Ship

Scenario

Example deployment: in four weeks an FDE builds a prototype that predicts which shipments will be late, using a Jupyter notebook, a CSV export and a personal API key. Executives love the demo and announce a go-live date three weeks out.

What you see in production

The customer's go-live review raises 23 findings: the job runs on the FDE's laptop, credentials are personal, there's no retry or monitoring, results are written to a shared drive, nobody at the customer can rerun it, and the model has never been evaluated on the last quarter's data. The go-live slips by two months, and the executive sponsor's trust slips with it.

The naive code

# notebook cell 14
API_KEY = "sk-live-…"                  # personal key
df = pd.read_csv("~/Downloads/shipments_march.csv")
df["late_risk"] = model.predict(df)
df.to_csv("/Volumes/shared/late_shipments.csv")   # the "integration"

Reproduce it

A go-live review is predictable; score the prototype against it as soon as the pilot starts:

REVIEW = ["runs on customer-approved infrastructure", "service credentials in a secret manager",
          "config outside code", "retries and idempotent reruns", "input validation and quarantine",
          "evaluated on recent real data", "monitoring and alerts", "runbook and named owner",
          "security review passed", "customer team trained"]

plan_by_week = {1: ["config outside code", "service credentials in a secret manager"],
                2: ["runs on customer-approved infrastructure", "retries and idempotent reruns", "security review passed"],
                3: ["input validation and quarantine", "evaluated on recent real data", "monitoring and alerts"],
                4: ["runbook and named owner", "customer team trained"]}

done = set()
print(f"prototype passes {len(done)}/{len(REVIEW)} review items")
for week, items in plan_by_week.items():
    done |= set(items)
    print(f"week {week}: {len(done)}/{len(REVIEW)}")
missing = [item for item in REVIEW if item not in done]
print("still missing:", missing or "nothing")

prototype passes 0/10 review items
week 1: 2/10
week 2: 5/10
week 3: 8/10
week 4: 10/10
still missing: nothing

Root cause

The prototype was optimised for speed of learning, which is right, but nobody made the production gap visible, so the go-live date was set from the demo, not from the work. Production requirements (security, operations, ownership) were discovered at the review instead of planned from the start.

The fix

1. Label prototypes as prototypes and, in the same meeting as the demo, show the path to production with its milestones.
2. Start long-lead items immediately: security review, service accounts, infrastructure approval.
3. Close the gap in thin slices: move the job to the customer's approved platform early, then add config, secrets, retries, validation, monitoring, evaluation and docs.
4. Name the customer-side owner who will run it after you leave, and train them during the pilot.
5. Keep the evaluation honest: test on the most recent data the model hasn't seen.

Trade-offs

• Productionising early slows the first demo slightly but prevents the much bigger slip later.
• Sometimes the right answer is to rebuild cleanly on the product rather than harden notebook code; decide explicitly.
• Executives want dates; give a range tied to the review checklist rather than a single optimistic date.

Detect it in production

• A shared readiness checklist with the customer, reviewed weekly.
• Count of review findings at go-live (target: none new).
• Time from pilot success to production, tracked across deployments.

Key points

• Prototypes prove value; production needs security, operations and ownership.
• Make the gap visible at the demo and start long-lead approvals immediately.
• Close the gap in thin slices with a named customer owner.