The Compass DeRose Guide to which characters are safe in URLs, email, and XML public identifiers

This page was written by Steven J. DeRose, and was last updated on 2005-10-20.

This table is from pages 88-89 of my The SGML FAQ book, which is really more about XML than SGML... The book is published by Kluwer Academic Publishers. Besides giving the historical and formal rationale that underlies XML, people tell me the SGML primer in Appendix A is a very accessible introduction to SGML, and perhaps the most compact.

I tried to put a lot of little touches like this in there, since I'm forever referring to such information myself, and it can be a pain to find; having such reference information close at hand can be pretty useful.

FPIs are used in SGML and XML to identify resources by name, not location: much like URNs. They use a limited character set, for portability. Other systems concerned for portability make similar restrictions.

URLs on the Web use a character set defined in IETF RFC 1738 "Uniform Resource Locators (URL)" (and several other RFCs that update or extend URLs). The characters allowed are A-Z, a-z, 0-9, space, plus these ten special characters:

     - $ _ . + ! * , ( )

Some additional characters (;/?:@=&) are reserved in some particular URL schemes but may be used otherwise. Some other characters are ruled out because they are used in other places in HTML files, even though they should not pose a conflict in URLs (for example, "<" and ">" pose no conflict so long as URLs are quoted as they should be). Other characters need to be expressed using a mechanism similar to SGML character references: "%" plus the hexadecimal code for the desired character.

The Text Encoding Initiative did extensive testing of what characters survive transfer through the many kinds of machines on the Internet without any special layers of re-encoding to hide them (say, as plain email content), and established a set of "safe" characters that is documented in the TEI Guidelines, Section 4.3. That set, called the ISO 646 subset (ISO 646 is the international standard the underlies ASCII), includes a-z, A-Z, 0-9, and these nineteen special characters:

     " % & ' ( ) * + , - . / : ; < = > ? _

A comparison of these sets is shown below. None of these sets permits characters outside of the printable 7-bit range of characters defined in ISO 646. "+" indicates permissible characters; "~" indicates characters that are only reserved in some URL schemes, and may otherwise be usable in URLs.

The names shown for characters below are taken from ISO 8859-1: Latin Alphabet Number 1 and ISO 646: Information Processing -- ISO 7-bit coded character set for information interchange (except that in those standards, they're all upper case). ISO 646 specifies that the number sign (decimal 35) may also represent the pound sterling sign. Likewise, decimal code 36 may represent either the dollar sign, or the "neutral currency sign".

Those characters shown in red must be escaped in certain situations when they occur in HTML, XML, or SGML files.

The "upper half", or characters from 128-255 (decimal), are less dependable -- many different "code pages" exist, assigning different characters to different places in that range. Neither Windows™ nor Mac™ generally uses the standard "Latin-1" set (defined by ISO 8859-1). In UTF-8 Unicode files, even those standard Latin-1 characters require special encoding. You can find the Windows character set here; the Mac character set here, and Latin-1 here. Microsoft provides a number of code pages here, and pointers to many other charts are here.

The list of HTML entities is defined here.

Unix jargon names for the various characters also exist.

IETF RFC 2152, "UTF-7: A Mail-Safe Transformation Format of Unicode", defines an emailable transformation of Unicode, and discusses issues of character safety in detail.

The Unicode Standard defines UTF-8, a byte-oriented encoding form of Unicode. For details, see Section 2.5 ÒEncoding FormsÓ and Section 3.9 "Unicode Encoding Forms".

About.com provides a page listing the safe characters to use in email, and has the excellent good sense to cite the TEI "ISO 646 subset" list. Bravo.

Back to home page of Steve DeRose or The Bible Technologies Group. or The Bible Technologies Group Working Groups. Or, contact me via email fix the punctuation).