XML in Production · Capstone Case Studies

Capstone: Postmortem — The Invoices That Lost Their Totals

Write the postmortem for an incident in which 3,400 invoices were sent to customers with a total of 0.00 after a supplier system upgrade.

Create a free account to save this page, take notes and track your progress. Sign in

Scenario

Example company: an invoicing service builds customer invoices from line items sent by an upstream ERP as XML. On Monday the ERP team upgrades to a new version that wraps amounts in a namespace (<amt:value xmlns:amt="urn:erp:amounts">) and sends empty <discount/> elements.

Timeline

06:00 Upgrade completes; the nightly invoice batch starts.
06:20 3,400 invoices are generated and emailed. Every total is 0.00; discounts are 0%.
08:45 Customer support receives the first complaints.
09:30 Batch job disabled; investigation begins.
12:10 Root cause found; corrected invoices regenerated with an apology email.

Facts

The parser code used findtext("amount") or "0" and float(findtext("discount") or 0).
No schema validation; no check that totals were non-zero.
The ERP team had sent sample files a week earlier; they weren't added to tests.

Deliverables

Root cause and contributing factors.
Why nothing stopped the bad invoices.
Immediate remediation.
Long-term fixes with owners.

Draft each deliverable before revealing the model answer and rubric.

Further reading: Namespaces in XML · XML Schema: xsi:nil

Deliverables

Root cause and contributing factors.

Model answer

Root cause: the new ERP version moved amounts into a namespace and started sending empty discounts; namespace-blind lookups returned None, and or "0" defaults turned missing and empty values into zero. Contributing: no contract (schema) validation, defaults that hide missing data, sample files not added to tests, and no coordination between the ERP upgrade and its consumers.

Rubric

Full marks: names both the namespace change and the silent defaults, plus process gaps.

Why nothing stopped the bad invoices.

Model answer

No validation step, no business-rule checks (an invoice total of 0.00 with non-zero lines, or every invoice in a batch with the same total, should fail), no canary (send a small batch first, or hold for review when anomalies appear), and no alerting on output distribution.

Rubric

Full marks: validation, sanity checks, canary or hold, distribution alerts.

Immediate remediation.

Model answer

Stop the batch, identify affected invoices, fix the parser to read the new namespace (supporting old and new), regenerate and re-send with an explanation, ensure no payments were taken at 0.00, and notify finance and support with a customer-facing statement.

Rubric

Full marks: containment, correction and communication, with financial checks.

Long-term fixes with owners.

Model answer

Integration team: XSD + Schematron validation of the ERP feed and explicit handling of absent/empty/nil. ERP team: publish versioned schemas and notify consumers through a change process; consumers add their sample files to CI. Invoicing team: business-rule assertions (non-zero totals, totals equal sum of lines), anomaly alerts on batch output, and a hold-for-review step when checks fail.

Rubric

Full marks: concrete fixes per team covering contract, validation, defaults and release process.

Open in the app to mark lessons complete, take notes and track your progress.