How to use Unicode code points, properties, blocks, and scripts in regular expressions

\u2122

This regex works in Python only when quoted as a Unicode string: u"\u2122".

\x{2122}

PCRE must be compiled with UTF-8 support; in PHP, turn on UTF-8 support with the /u pattern modifier. Ruby 1.8 does not support Unicode regular exp​ressions.

\p{Sc}

PCRE must be compiled with UTF-8 support; in PHP, turn on UTF-8 support with the /u pattern modifier. Javascript and Python do not support Unicode properties. Ruby 1.8 does not support Unicode regular exp​ressions.

\p{IsGreekExtended}

\p{InGreekExtended}

Javascript, PCRE, Python, and Ruby do not support Unicode blocks.

\p{Greek}

Unicode script support requires PCRE 6.5 or later, and PCRE must be compiled with UTF-8 support. In PHP, turn on UTF-8 support with the /u pattern modifier. .NET, Javascript, and Python do not support Unicode properties. Ruby 1.8 does not support Unicode regular exp​ressions.

\X

PCRE and Perl have a dedicated token for matching graphemes, but also support the workaround syntax using Unicode properties.

\P{M}\p{M}*

PCRE must be compiled with UTF-8 support; in PHP, turn on UTF-8 support with the /u pattern modifier. Javascript and Python do not support Unicode properties. Ruby 1.8 does not support Unicode regular exp​ressions.

A code point is one entry in the Unicode character database. A code point is not the same as a character, depending on the meaning you give to “character.” What appears as a character on screen is called a grapheme in Unicode.

The Unicode code point U+2122 represents the “trademark sign” character. You can match this with \u2122 or \x{2122}, depending on the regex flavor you’re working with.

The \u syntax requires exactly four hexadecimal digits. This means you can only use it for Unicode code points U+0000 through U+FFFF. The \x syntax allows any number of hexadecimal digits, supporting all code points U+000000 through U+10FFFF. You can match U+00E0 with \x{E0} or \x{00E0}. Code points U+100000 are used very infrequently, and are poorly supported by fonts and operating systems.

Code points can be used inside and outside character classes.

Each Unicode code point has exactly one Unicode property, or fits into a single Unicode category. These terms mean the same thing. There are 30 Unicode categories, grouped into 7 super-categories:

\p{Ll} matches a single code point that has the Ll, or “lowercase letter,” property. \p{L} is a quick way of writing [\p{Ll}\p{Lu}\p{Lt}\p{Lm}\p{Lo}] that matches a single code point in any of the “letter” categories.

\P is the negated version of \p. \P{Ll} matches a single code point that does not have the Ll property. \P{L} matches a single code point that does not have any of the “letter” properties. This is not the same as [\P{Ll}\P{Lu}\P{Lt}\P{Lm}\P{Lo}], which matches all code points. \P{Ll} matches the code points with the Lu property (and every other property except Ll), whereas \P{Lu} includes the Ll code points. Combining just these two in a code point class already matches all possible code points.

The Unicode character database divides all the code points into blocks. Each block consists of a single range of code points. The code points U+0000 through U+FFFF are divided into 105 blocks:

A Unicode block is a single, contiguous range of code points. Although many blocks have the names of Unicode scripts and Unicode categories, they do not correspond 100% with them. The name of a block only indicates its primary use.

The Currency block does not include the dollar and yen symbols. Those are found in the Basic_Latin and Latin-1_Supplement blocks, for historical reasons. Both do have the Currency Symbol property. To match any currency symbol, use \p{Sc} instead of \p{InCurrency}.

Most blocks include unassigned code points, which are covered by the property \p{Cn}. No other Unicode property, and none of the Unicode scripts, include unassigned code points.

The \p{InBlockName} syntax works with .NET and Perl. Java uses the \p{IsBlockName} syntax.

Perl also supports the Is variant, but we recommend you stick with the In syntax, to avoid confusion with Unicode scripts. For scripts, Perl supports \p{Script} and \p{IsScript}, but not \p{InScript}.

Each Unicode code point, except unassigned ones, is part of exactly one Unicode script. Unassigned code points are not part of any script. The assigned code points up to U+FFFF are assigned to these scripts:

A script is a group of code points used by a particular human writing system. Some scripts, such as Thai, correspond with a single human language. Other scripts, such as Latin, span multiple languages. Some languages are composed of multiple scripts. For instance, there is no Japanese Unicode script; instead, Unicode offers the Hiragana, Katakana, Han, and Latin scripts that Japanese documents are usually composed of.

We listed the Common script first, out of alphabetical order. This script contains all sorts of characters that are common to a wide range of scripts, such as punctuation, whitespace, and miscellaneous symbols.

The difference between code points and characters comes into play when there are combining marks. The Unicode code point U+0061 is “Latin small letter a,” whereas U+00E0 is “Latin small letter a with grave accent.” Both represent what most people would describe as a character.

U+0300 is the “combining grave accent” combining mark. It can be used sensibly only after a letter. A string consisting of the Unicode code points U+0061 U+0300 will be displayed as à, just like U+00E0. The combining mark U+0300 is displayed on top of the character U+0061.

The reason for these two different ways of displaying an accented letter is that many historical character sets encode “a with grave accent” as a single character. Unicode’s designers thought it would be useful to have a one-on-one mapping with popular legacy character sets, in addition to the Unicode way of separating marks and base letters, which makes arbitrary combinations not supported by legacy character sets possible.

What matters to you as a regex user is that all regex flavors discussed in this book operate on code points rather than graphical characters. When we say that the regular exp​ression . matches a single character, it really matches just a single code point. If your subject text consists of the two code points U+0061 U+0300, which can be represented as the string literal "\u0061\u0300" in a programming language such as Java, the dot will match only the code point U+0061, or a, without the accent U+0300. The regex .. will match both.

Perl and PCRE offer a special regex token \X, which matches any single Unicode grapheme. Essentially, it is the Unicode version of the venerable dot. It matches any Unicode code point that is not a combining mark, along with all the combining marks that follow it, if any. \P{M}\p{M}* does the same thing using the Unicode property syntax. \X will find two matches in the text àà, regardless of how it is encoded. If it is encoded as "\u00E0\u0061\u0300" the first match is "\u00E0", and the second "\u0061\u0300".

The uppercase \P is the negated variant of the lowercase \p. For instance, \P{Sc} matches any character that does not have the “Currency Symbol” Unicode property. \P is supported by all flavors that support \p, and for all the properties, block, and scripts that they support.

All flavors allow all the \u, \x, \p, and \P tokens they support to be used inside character classes. The character represented by the code point, or the characters in the category, block, or script, are then added to the character class. For instance, you could match a character that is either an opening quote (initial punctuation property), a closing quote (final punctuation property), or the trademark symbol (U+2122) with:

[\p{Pi}\p{Pf}\x{2122}]

If your regular exp​ression flavor does not support Unicode categories, blocks, or scripts, you can list the characters that are in the category, block, or script in a character class. For blocks this is very easy: each block is simply a range between two code points. The Greek Extended block comprises the characters U+1F00 to U+1FFF:

[\u1F00-\u1FFF]

[\x{1F00}-\x{1FFF}]

For most categories and many scripts, the equivalent character class is a long list of individual code points and short ranges. The characters that comprise each category and many of the scripts are scattered throughout the Unicode table. This is the Greek script:

[\u0370-\u0373\u0375\u0376-\u0377\u037A\u037B-\u037D\u0384\u0386↵

\u0388-\u038A\u038C\u038E-\u03A1\u03A3-\u03E1\u03F0-\u03F5\u03F6↵

\u03F7-\u03FF\u1D26-\u1D2A\u1D5D-\u1D61\u1D66-\u1D6A\u1DBF\u1F00-\u1F15↵

\u1F18-\u1F1D\u1F20-\u1F45\u1F48-\u1F4D\u1F50-\u1F57\u1F59\u1F5B\u1F5D↵

\u1F5F-\u1F7D\u1F80-\u1FB4\u1FB6-\u1FBC\u1FBD\u1FBE\u1FBF-\u1FC1↵

\u1FC2-\u1FC4\u1FC6-\u1FCC\u1FCD-\u1FCF\u1FD0-\u1FD3\u1FD6-\u1FDB↵

\u1FDD-\u1FDF\u1FE0-\u1FEC\u1FED-\u1FEF\u1FF2-\u1FF4\u1FF6-\u1FFC↵

\u1FFD-\u1FFE\u2126]

We built this regular exp​ression by copying the listing for the Greek script from http://www.unicode.o...ATA/Scripts.txt, searching and replacing with three regular exp​ressions:

This may seem like a lot of work, but it actually took Jan less than a minute. Writing the description took much longer. Doing this for the \x{} syntax is just as easy:

The results are:

[\x{0370}-\x{0373}\x{0375}\x{0376}-\x{0377}\x{037A}\x{037B}-\x{037D}↵

\x{0384}\x{0386}\x{0388}-\x{038A}\x{038C}\x{038E}-\x{03A1}↵

\x{03A3}-\x{03E1}\x{03F0}-\x{03F5}\x{03F6}\x{03F7}-\x{03FF}↵

\x{1D26}-\x{1D2A}\x{1D5D}-\x{1D61}\x{1D66}-\x{1D6A}\x{1DBF}↵

\x{1F00}-\x{1F15}\x{1F18}-\x{1F1D}\x{1F20}-\x{1F45}\x{1F48}-\x{1F4D}↵

\x{1F50}-\x{1F57}\x{1F59}\x{1F5B}\x{1F5D}\x{1F5F}-\x{1F7D}↵

\x{1F80}-\x{1FB4}\x{1FB6}-\x{1FBC}\x{1FBD}\x{1FBE}\x{1FBF}-\x{1FC1}↵

\x{1FC2}-\x{1FC4}\x{1FC6}-\x{1FCC}\x{1FCD}-\x{1FCF}\x{1FD0}-\x{1FD3}↵

\x{1FD6}-\x{1FDB}\x{1FDD}-\x{1FDF}\x{1FE0}-\x{1FEC}\x{1FED}-\x{1FEF}↵

\x{1FF2}-\x{1FF4}\x{1FF6}-\x{1FFC}\x{1FFD}-\x{1FFE}\x{2126}↵

\x{10140}-\x{10174}\x{10175}-\x{10178}\x{10179}-\x{10189}↵

\x{1018A}\x{1D200}-\x{1D241}\x{1D242}-\x{1D244}\x{1D245}]