August 5, 2022

Unicode: flag "u" and class \p{...}

JavaScript uses Unicode encoding for strings. Most characters are encoded with 2 bytes, but that allows to represent at most 65536 characters.

That range is not big enough to encode all possible characters, that’s why some rare characters are encoded with 4 bytes, for instance like 𝒳 (mathematical X) or 😄 (a smile), some hieroglyphs and so on.

Here are the Unicode values of some characters:

Character Unicode Bytes count in Unicode
a 0x0061 2
0x2248 2
𝒳 0x1d4b3 4
𝒴 0x1d4b4 4
😄 0x1f604 4

So characters like a and occupy 2 bytes, while codes for 𝒳, 𝒴 and 😄 are longer, they have 4 bytes.

Long time ago, when JavaScript language was created, Unicode encoding was simpler: there were no 4-byte characters. So, some language features still handle them incorrectly.

For instance, length thinks that here are two characters: