Forward Deployed Engineering in Practice · Capstone Case Studies
Capstone: Postmortem — The Pilot That Stalled
A 10-week pilot ended without a production decision. Reconstruct what went wrong and how a forward deployed engineer could have changed the outcome.
Scenario
Example customer: a logistics company piloted a vendor's AI route-exception assistant with one regional operations team. The pilot was planned for 10 weeks. In week 12 the customer's COO declined to proceed: "Nice technology, unclear value."
Evidence
- Kickoff deck listed goals as "improve operations with AI"; no metric or baseline was recorded.
- Security review started in week 4 and finished in week 9; real data arrived in week 9.
- Weeks 1-8 used synthetic data; demos impressed the sponsor's team but operators found the scenarios unrealistic.
- The FDE built 19 requested features, including four dashboards; the core exception workflow shipped in week 10.
- Weekly status emails listed completed tasks; no risks were raised before week 8.
- The sponsor (a regional director) changed roles in week 7; nobody re-engaged the COO.
- Three other customers had asked the vendor's product team for the same data connector the FDE built by hand.
Deliverables
- The main reasons the pilot stalled.
- What should have happened in the first two weeks.
- How to recover the account now, if possible.
- What the vendor should change for future pilots.
Draft each deliverable before revealing the model answer and rubric.
Further reading: Shape Up · NIST AI Risk Management Framework
Deliverables
Why did the pilot stall?
Model answer
- No measurable success criteria or baseline, so value couldn't be demonstrated.
- Security review started late and sat on the critical path; real data arrived in week 9.
- Effort spread across 19 requests instead of the core workflow.
- Status reporting hid risks; the sponsor change wasn't managed and the executive decision-maker was never engaged.
Rubric
Full marks: identifies missing success criteria, late critical-path dependency, scope dilution and stakeholder management failures.
What should have happened in the first two weeks?
Model answer
Shadow operators to find the top exception types and their cost; agree one or two metrics with a baseline (for example, time to resolve route exceptions, missed delivery windows); write a one-page scope with out-of-scope items; submit security and data-access requests on day one; map stakeholders including the COO as decision-maker; set a weekly status with risks and asks; plan an evaluation using real historical exceptions.
Rubric
Full marks: concrete discovery, metrics with baseline, early approvals, stakeholder mapping and risk-based reporting.
How could the account be recovered?
Model answer
Propose a short, focused extension (4 weeks) with explicit success criteria and the baseline measured from historical data; run the core workflow with real data for the original team; meet the new sponsor and the COO with a one-page value case and the measurement plan; drop non-core features from scope; agree the production decision criteria upfront.
Rubric
Full marks: specific, time-boxed recovery tied to measurable value and the actual decision-maker.
What should the vendor change for future pilots?
Model answer
A pilot playbook: success criteria and baseline required at kickoff; standard security package and day-one access requests; real-data milestone by week 3; triage process for requests; weekly risk-based status; executive check-ins at kickoff, midpoint and end; FDE field feedback flowing to product (the connector three customers needed should become a product feature).
Rubric
Full marks: systemic changes across process, product feedback and executive engagement, not just individual effort.
Open in the app to mark lessons complete, take notes and track your progress.