XML in Production · Capstone Case Studies
Capstone: A Safe, Scalable XML Ingestion Pipeline
Design the pipeline that receives, validates, transforms and loads XML from 200 partners, from 2 KB messages to 6 GB nightly files.
Scenario
Example company: a distributor receives product, price and order data from 200 suppliers. Messages arrive by SFTP (nightly files up to 6 GB), HTTPS POST (small orders, around 50 per second at peak) and a few SOAP services. Today one monolithic script parses everything with defaults, and a recent penetration test found XXE in the upload endpoint.
Requirements
- Safe against XXE, entity expansion and oversized input.
- Large files processed in bounded memory; small messages with low latency.
- Validation against each partner's contract version, with clear rejection reasons.
- No message lost: failures are quarantined and can be replayed.
- Observability per partner: volume, latency, rejection rate.
Deliverables
- The parsing and security baseline.
- Validation and versioning strategy.
- The large-file path.
- Error handling, quarantine and replay.
- Monitoring and alerting.
Hints
- One shared parser factory beats 200 careful developers.
- Validate structure with XSD, business rules with Schematron.
- Stream large files; validate per record.
Draft each deliverable before revealing the model answer and rubric.
Further reading: OWASP XXE Prevention Cheat Sheet · lxml: iterparse
Deliverables
Parsing and security baseline.
Model answer
A single library module creates every parser: DTDs and external entities disabled (lxml resolve_entities=False, no_network=True, load_dtd=False; Java disallow-doctype-decl), no huge_tree, body size limits at the edge (for example 5 MB for HTTPS), a parse timeout and per-worker memory limits. A CI rule (Semgrep) forbids constructing parsers elsewhere. Parsing services have no outbound internet access.
Rubric
Full marks: centralised safe factory, size/time/memory limits, enforcement in CI, egress restrictions.
Validation and versioning strategy.
Model answer
A registry maps partner and message type to schema version (namespace or version attribute selects it). XSD validates structure and types, with extension points (xs:any lax) so additive changes pass; Schematron validates business rules (totals, dates, currency codes). Each partner has sample files per version in CI, and published partner schemas are pulled nightly. Old versions are supported for a deprecation window with usage metrics.
Rubric
Full marks: explicit versions, XSD + Schematron split, tolerant extension points, CI contract tests.
The large-file path.
Model answer
Files land in object storage; a worker streams them with iterparse/XMLReader/StAX, expands one record at a time, validates the record against the schema fragment, transforms it, and writes in batches of 1,000 with a checkpoint (file id plus record offset) so a crash resumes. Memory stays flat regardless of size; throughput is scaled by processing different files in parallel.
Rubric
Full marks: streaming with cleanup, per-record validation, batching and resumable checkpoints.
Error handling, quarantine and replay.
Model answer
Invalid messages go to a quarantine store with the reason, schema version and partner; nothing is dropped. Partners get a structured rejection (code, path, message) synchronously for HTTPS and in a daily report for files. Operators can fix configuration and replay a quarantined message or a whole file idempotently (keys on partner message ids).
Rubric
Full marks: quarantine with reasons, partner feedback, idempotent replay.
Monitoring and alerting.
Model answer
Per partner and message type: received, accepted, rejected (by reason), latency, file size and record counts. Alerts on zero volume when volume is expected, rejection-rate spikes, sudden drops in record counts (namespace or contract changes), parser errors mentioning DOCTYPE/ENTITY (attacks), and worker OOMs or timeouts.
Rubric
Full marks: volume and quality metrics per partner, anomaly alerts covering silent failures and attacks.
Open in the app to mark lessons complete, take notes and track your progress.