Invalid Characters in XML

1. Overview

XML (eXtensible Markup Language) is one of the most widely used formats for storing and transporting data across various platforms and applications. However, despite its robustness, XML is not immune to issues, and one such challenge arises from handling invalid characters within XML documents.

In this article, we’ll look at different invalid characters and how we can handle them in XML processing.

2. Valid Characters in XML

The XML specification defines the characters that are allowed in element content and attribute values. According to the XML 1.0 specification, the acceptable characters are listed below. XML considers any character outside of these ranges as invalid characters:

Description	Range	Examples
Tab character (Horizontal Tab)	9 (TAB)	\t
Line Feed character (New Line)	10 (LF)	\n
Carriage Return character (Return to the beginning of the line)	13 (CR)	\r
Characters in the Basic Multilingual Plane (BMP), excluding surrogate blocks	32 to 55295	A, b, &, 1, α (Greek letter alpha)
Characters in the Supplementary Private Use Area-A (SMP), excluding surrogate blocks	57344 to 65533	(Smiling face), (Party popper)
Characters beyond the BMP in the Supplementary Planes	65536 to 1114111	(Globe with meridians), (Rocket)

Note: In Unicode, we use surrogate blocks as specific ranges of code points in UTF-16 encoding to represent characters beyond the Basic Multilingual Plane.

3. XML 1.1 and Handling Invalid Characters

XML 1.1, introduced as an update to XML 1.0, provides additional flexibility and support for a broader range of characters, including characters from the entire Unicode character set. It allows characters in the range 1-31 (except for TAB, LF, and CR) and certain control characters such as NEL (Next Line, Unicode 0x0085).

4. Invalid Characters in XML

Invalid characters in XML typically fall into two categories:

4.1. Reserved Characters

XML reserves certain characters for specific purposes within its syntax, such as <, >, &, “, and ‘. When these characters appear within the context of an XML element without proper encoding, they can disrupt the parsing process and render the XML document invalid. Let’s see a code example where we provide an  invalid character:

@Test
void givenXml_whenReservedCharacters_thenThrowException() {
    String invalidXmlString = "<?xml version=\"1.1\" encoding=\"UTF-8\"?><root><name>John & Doe</name></root>";
    assertThrowsExactly(SAXParseException.class, () -> parseXmlString(invalidXmlString));
}

We should properly escape reserved characters using predefined character entities. For instance:

• < should be encoded as &lt;
• > should be encoded as &gt;
• & should be encoded as &amp;
• “ should be encoded as &quot;
• ‘ should be encoded as &apos;

We can test it by executing the below test:

@Test
void givenXml_whenReservedCharactersEscaped_thenSuccess() {
    String validXmlString = "<?xml version=\"1.1\" encoding=\"UTF-8\"?><root><name>John &amp; Doe</name></root>";
    assertDoesNotThrow(() -> {
        Document document = parseXmlString(validXmlString);
        assertNotNull(document);
        assertEquals("John & Doe", document.getElementsByTagName("name").item(0).getTextContent());
    });
}

Another method to handle reserved characters in XML is by utilizing the CDATA Section. It serves as a means to encapsulate blocks of text that may contain characters otherwise interpreted as markup:

@Test
void givenXml_whenUsingCdataForReservedCharacters_thenSuccess() {
    String validXmlString = "<?xml version=\"1.1\" encoding=\"UTF-8\"?><root><name><![CDATA[John & Doe]]></name></root>";
    assertDoesNotThrow(() -> {
        Document document = parseXmlString(validXmlString);
        assertNotNull(document);
        assertEquals("John & Doe", document.getElementsByTagName("name").item(0).getTextContent());
    });
}

4.2. Unicode Characters

XML documents are encoded using Unicode, which supports a vast range of characters from different languages and scripts. While Unicode offers extensive coverage, it also includes characters that may not be compatible with XML’s encoding standards, leading to parsing errors.

Let’s examine the following test scenario, where we incorporate a record separator within XML. Unicode represents the record separator as \u001E:

@Test
void givenXml_whenUnicodeCharacters_thenThrowException() {
    String invalidXmlString = "<?xml version=\"1.1\" encoding=\"UTF-8\"?><root><name>John \u001E Doe</name></root>";
    assertThrowsExactly(SAXParseException.class, () -> parseXmlString(invalidXmlString));
}

The character has an ASCII value of 30, which is outside the accepted range. Hence, the test to parse it will fail. To handle non-ASCII characters correctly, we should encode them using Unicode schemes like UTF-8 or UTF-16.

This ensures compatibility across different platforms and avoids data corruption issues. Let’s now execute the below test with proper encoding:

@Test
void givenXml_whenUnicodeCharactersEscaped_thenSuccess() {
    String validXmlString = "<?xml version=\"1.1\" encoding=\"UTF-8\"?><root><name>John &#x1E; Doe</name></root>";
    assertDoesNotThrow(() -> {
        Document document = parseXmlString(validXmlString);
        assertNotNull(document);
        assertEquals("John \u001E Doe", document.getElementsByTagName("name").item(0).getTextContent());
    });
}

5. Conclusion

In this article, we looked at the different invalid characters in XML and how we can efficiently handle them. By understanding the causes of invalid characters and employing appropriate strategies for handling them, developers can ensure the robustness and reliability of their XML processing pipelines. 

As always, the full source code is available over on GitHub.