How do you parse a 10 GB XML file in Python with constant memory?

Use lxml (or ElementTree) iterparse with events=('end',) and a tag filter; process each element, then call clear() and delete earlier siblings so the tree never grows.

What are the PHP and Java equivalents?

PHP: XMLReader, reading node by node and expanding one record at a time. Java: StAX (XMLStreamReader) or SAX.

The 4 GB Product Feed — XML in Production

Scenario

Example team: a marketplace imports a supplier's full catalogue every night. The feed was 80 MB when the import was written; after the supplier added image metadata and variants, it reached 4 GB.

What you see in production

The import container is OOM-killed (exit code 137) at about 30% progress. Raising the memory limit to 16 GB "fixes" it for two weeks. Then the feed grows again.

The naive code

doc = etree.parse("catalogue.xml")                 # the whole tree in memory
for product in doc.getroot().iter("product"):
    upsert(product)

Reproduce it

Element counts stand in for memory (a parsed element costs hundreds of bytes in lxml):

Elements held in memoryTry it yourself

import io
from lxml import etree

feed = "<catalogue>" + "".join(
    f'<product sku="P{i}"><name>Item {i}</name><price>{i % 50 + 1}.99</price></product>' for i in range(5000)
) + "</catalogue>"

# Tree parsing: everything stays in memory
tree = etree.fromstring(feed.encode())
print("tree: elements in memory at the end:", sum(1 for _ in tree.iter()))

# Streaming without cleanup: still builds the full tree
root = None
for _, el in etree.iterparse(io.BytesIO(feed.encode()), events=("end",), tag="product"):
    root = el.getparent()
print("iterparse, no clear(): elements kept:", sum(1 for _ in root.iter()))

# Streaming with cleanup: memory stays flat
peak, total = 0, 0.0
for _, el in etree.iterparse(io.BytesIO(feed.encode()), events=("end",), tag="product"):
    total += float(el.findtext("price"))
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]
    peak = max(peak, sum(1 for _ in el.getparent().iter()))
print("iterparse + clear(): peak elements kept:", peak, "| total price:", round(total, 2))

import io
from lxml import etree

feed = "<catalogue>" + "".join(
    f'<product sku="P{i}"><name>Item {i}</name><price>{i % 50 + 1}.99</price></product>' for i in range(5000)
) + "</catalogue>"

# Tree parsing: everything stays in memory
tree = etree.fromstring(feed.encode())
print("tree: elements in memory at the end:", sum(1 for _ in tree.iter()))

# Streaming without cleanup: still builds the full tree
root = None
for _, el in etree.iterparse(io.BytesIO(feed.encode()), events=("end",), tag="product"):
    root = el.getparent()
print("iterparse, no clear(): elements kept:", sum(1 for _ in root.iter()))

# Streaming with cleanup: memory stays flat
peak, total = 0, 0.0
for _, el in etree.iterparse(io.BytesIO(feed.encode()), events=("end",), tag="product"):
    total += float(el.findtext("price"))
    el.clear()
    while el.getprevious() is not None:
        del el.getparent()[0]
    peak = max(peak, sum(1 for _ in el.getparent().iter()))
print("iterparse + clear(): peak elements kept:", peak, "| total price:", round(total, 2))

Output

tree: elements in memory at the end: 15001
iterparse, no clear(): elements kept: 15001
iterparse + clear(): peak elements kept: 1397 | total price: 132450.0

Root cause

Tree APIs (DOM, ElementTree, SimpleXML) build the whole document in memory, typically 5 to 10 times the file size. Memory grows with the input, so any limit eventually breaks. iterparse alone doesn't fix it: it still builds the tree unless you free processed elements.

The fix

Stream: lxml/ElementTree iterparse with clear() and removal of finished siblings, PHP XMLReader, Java StAX or SAX.
Process in batches (commit every 1,000 products) so a failure can resume.
If you need XPath on each record, expand one record at a time (readOuterXml() in PHP, etree.fromstring of a fragment).
Ask for incremental (delta) feeds when possible.

Trade-offs

Streaming code is more stateful and can't look ahead or back without buffering.
Validation of a whole multi-GB document needs a streaming validator (libxml2 XMLReader with a schema, Xerces) or per-record validation.

Detect it in production

Container memory metrics and OOM kills (exit 137) for import jobs.
Track feed size over time and alert when it doubles.
Load-test imports with a feed 10x the current size.

Key points

Tree parsers use memory proportional to the document; large feeds need streaming.
With iterparse, clear processed elements or you gain nothing.
Batch, checkpoint and watch feed growth.