Why should you pass bytes rather than a decoded string to an XML parser?

The XML declaration tells the parser which encoding the bytes use. If you decode first, you guess the encoding, and if you re-encode the text, the declaration may no longer match the bytes.

What does 'MÃ¼ller' tell you about a bug?

UTF-8 bytes were decoded as Latin-1/Windows-1252: the two-byte UTF-8 sequence for ü became two characters. Find the step that decoded with the wrong encoding.

Müller Became MÃ¼ller — XML in Production

Scenario

Example team: a CRM sync service downloads customer records as XML over HTTP, converts them to strings, adds a field, and posts the result to a billing system. The upstream system sends ISO-8859-1 and says so in the XML declaration.

What you see in production

Customer "Müller" appears as "MÃ¼ller" in invoices; "Zoë" becomes "ZoÃ«". Support tickets arrive from Germany and France. Some records fail to parse at all with "Invalid bytes in character encoding".

The naive code

text = response.content.decode("utf-8", errors="replace")   # guess the encoding
doc = etree.fromstring(text.encode("utf-8"))                 # declaration now lies

Reproduce it

Decoding with the wrong encodingTry it yourself

from lxml import etree

payload = '<?xml version="1.0" encoding="ISO-8859-1"?><customer>Müller &amp; Zoë</customer>'.encode("iso-8859-1")

# Right: give the parser the bytes; it reads the declaration
print("bytes to parser:", etree.fromstring(payload).text)

# Wrong 1: decode as UTF-8 first (bad bytes are replaced)
guessed = payload.decode("utf-8", errors="replace")
print("decoded as UTF-8:", guessed[guessed.index("<customer>") + 10:guessed.index("</customer>")])

# Wrong 2: decode as Latin-1, re-encode as UTF-8, but keep the old declaration -> classic mojibake
text = payload.decode("iso-8859-1").encode("utf-8")
print("declaration now lies:", etree.fromstring(text).text)

Output

bytes to parser: Müller & Zoë
decoded as UTF-8: M�ller &amp; Zo�
declaration now lies: MÃ¼ller & ZoÃ«

Root cause

XML carries its own encoding in the declaration, and the parser is the component that should apply it. Decoding the bytes yourself means guessing. Re-encoding without updating the declaration then makes the parser decode UTF-8 bytes as Latin-1, turning each two-byte UTF-8 character into two wrong characters (the "Ã" pattern).

The fix

Pass bytes to the XML parser (etree.fromstring(response.content)), not decoded strings.
Write output as UTF-8 with a matching declaration (etree.tostring(doc, encoding="UTF-8", xml_declaration=True)).
If an HTTP Content-Type charset disagrees with the declaration, decide on one rule (for application/xml, the charset parameter wins per RFC 7303) and log mismatches.
Add test fixtures with accented and non-Latin text (é, ß, ł, 中文) to every integration.

Trade-offs

Some legacy partners declare one encoding and send another. A repair step may be needed, but quarantine those messages instead of silently "fixing" everything.
Normalising everything to UTF-8 internally is simpler than supporting many encodings end to end.

Detect it in production

Count parser errors about encodings per partner.

Key points

Let the XML parser decode bytes using the declaration; never decode first.
Output UTF-8 with a correct declaration.
Test every integration with non-ASCII data.