Escaping, CDATA and Encoding — XML Tutorial

The five predefined entities

Character	Escape	Required?
`<`	`<`	Always in text and attribute values
`&`	`&`	Always
`>`	`>`	Only in `]]>`; escaping it is harmless
`"`	`"`	Inside a `"`-quoted attribute
`'`	`'`	Inside a `'`-quoted attribute

Character references insert any Unicode character by number: é or é both mean é. Named entities such as   or © are HTML, not XML; in XML they're errors unless a DTD defines them.

Let the library escape for you

Build XML with an API and every value is escaped correctly, including the awkward ones:

Values that need escapingTry it yourself

from lxml import etree

row = etree.Element("product", name='Fish & "Chips"')
row.text = "Price < 5 & tasty"
print(etree.tostring(row).decode())
print(etree.fromstring(etree.tostring(row)).get("name"))

Output

<product name="Fish &amp; &quot;Chips&quot;">Price &lt; 5 &amp; tasty</product>
Fish & "Chips"

Values that need escapingTry it yourself

<?php
$doc = new DOMDocument();
$row = $doc->createElement("product");
$row->setAttribute("name", 'Fish & "Chips"');
$row->appendChild($doc->createTextNode("Price < 5 & tasty"));
$doc->appendChild($row);
echo $doc->saveXML($row), "\n";
echo simplexml_load_string($doc->saveXML($row))['name'], "\n";

Output

<product name="Fish &amp; &quot;Chips&quot;">Price &lt; 5 &amp; tasty</product>
Fish & "Chips"

Values that need escaping

import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.*;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.*;
import java.io.*;

public class Main {
    public static void main(String[] args) throws Exception {
        var builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        Document doc = builder.newDocument();
        Element row = doc.createElement("product");
        row.setAttribute("name", "Fish & \"Chips\"");
        row.setTextContent("Price < 5 & tasty");
        doc.appendChild(row);
        Transformer t = TransformerFactory.newInstance().newTransformer();
        t.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
        StringWriter out = new StringWriter();
        t.transform(new DOMSource(doc), new StreamResult(out));
        System.out.println(out);
        Element back = builder.parse(new ByteArrayInputStream(out.toString().getBytes("UTF-8"))).getDocumentElement();
        System.out.println(back.getAttribute("name"));
    }
}

import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.*;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.*;
import java.io.*;

public class Main {
    public static void main(String[] args) throws Exception {
        var builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        Document doc = builder.newDocument();
        Element row = doc.createElement("product");
        row.setAttribute("name", "Fish & \"Chips\"");
        row.setTextContent("Price < 5 & tasty");
        doc.appendChild(row);
        Transformer t = TransformerFactory.newInstance().newTransformer();
        t.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
        StringWriter out = new StringWriter();
        t.transform(new DOMSource(doc), new StreamResult(out));
        System.out.println(out);
        Element back = builder.parse(new ByteArrayInputStream(out.toString().getBytes("UTF-8"))).getDocumentElement();
        System.out.println(back.getAttribute("name"));
    }
}

Output

<product name="Fish &amp; &quot;Chips&quot;">Price &lt; 5 &amp; tasty</product>
Fish & "Chips"

Run this on your computer — see “Get Started” for setup.

Building XML with string concatenation ("<name>" + value + "</name>") breaks as soon as a value contains < or &, and it's an injection risk: a value like </name><admin>true</admin><name> changes the structure.

CDATA sections

Inside <![CDATA[ … ]]> everything is literal text, which is handy for code samples or embedded HTML:

from lxml import etree doc = etree.fromstring(b"<snippet><![CDATA[if (a < b && c > d) { alert('hi'); }]]></snippet>") print(doc.text) print(etree.tostring(doc).decode()) # serialised again with normal escaping

CDATA is only a writing convenience; after parsing, the text is the same as if it had been escaped. The only sequence a CDATA section can't contain is ]]>.

Encodings

XML parsers must support UTF-8 and UTF-16; UTF-8 is the default and the right choice. Declare anything else (<?xml version="1.0" encoding="ISO-8859-1"?>), and make sure the bytes really are in that encoding. Parse bytes, not decoded strings, so the parser can honour the declaration:

from lxml import etree latin1 = '<?xml version="1.0" encoding="ISO-8859-1"?><city>Zürich</city>'.encode("iso-8859-1") print(latin1[-12:]) # the raw bytes: ü is one byte, 0xfc print(etree.fromstring(latin1).text) # decoded correctly thanks to the declaration try: etree.fromstring('<city>Zürich</city>'.encode("iso-8859-1")) # no declaration: UTF-8 assumed except etree.XMLSyntaxError as e: print("error:", e.msg)

The five predefined entities

Let the library escape for you

CDATA sections

Encodings

Test yourself