Brian's Website


You never have to think about Unicode until you do.

I recently read this blog post: The Absolute Minimum Every Software Developer Must Know About Unicode in 2023 (Still No Excuses!). It’s a nod to a similarly titled 2003 post by Joel Spolsky. And if you are a Go programmer, you may have encountered this blog post, which explains how Go handles strings (it also references the Spolsky post).

I highly recommend all three - the third even if you aren’t a Go programmer.

The premise is that we (software engineers) often take strings for granted, but there are fundamentals to text encoding that are worth grokking. The majority of the time you don’t need to deal with these details. But, like many topics, eventually the edge cases come and demand a deeper understanding.

Key Take-Aways


Unicode is a standard that maps all the characters in the world to “code points”.

Code points are just numbers, usually written as hexadecimal digits with a U+ prefix, e.g.:

NameCharacterCode PointDecimal Value
Latin Capital Letter AAU+004165
Rightwards arrowU+21928594
Dog face🐶U+1F436128054

Unicode is an evolving standard. Its codespace covers U+0000 to U+10FFFF, about 1.1M code points, of which currently only ~15% are defined. The Unicode Consortium periodically releases new versions which include new character mappings.


Code points are somewhat theoretical: they just map characters to numbers. But there are different ways for computers to store these values in memory.

Character Encodings are the concrete implementations for storing strings in memory.

UTF-8 is the most popular encoding (98% of web pages according to the post). UTF-8 is a variable-length encoding, so characters are represented by sequences of 1 to 4 bytes depending on their value.

First code pointLast code pointByte 1Byte 2Byte 3Byte 4

The first 128 code points (U+0000 - U+007F) are stored in 1-byte and are equivalent to ASCII, making UTF-8 backward compatible with ASCII. Higher code points use 2, 3, or 4 bytes according to the table above. The byte prefixes indicate whether a byte is the leading, or non-leading, byte in a 1-, 2-, 3- or 4-byte sequence.

UTF-16 is another encoding. Despite being less popular than UTF-8, UTF-16 is important for web developers because it’s what JavaScript uses.

UTF-16 encodes strings as sequences of 16-bit “code units”. Because some code points are larger than 16-bits, UTF-16 uses a technique called “surrogate pairs”, which, as the name suggests, are pairs of 16-bit code units that represent a single character. Thus UTF-8 and UTF-16 are both variable length, but the implementation is different.

To illustrate with some JavaScript:

const message = 'hi 😍';
message.length; // => 5
message.split(''); // => [ 'h', 'i', ' ', '\ud83d', '\ude0d' ]

// iterate over UTF-16 code units
for (let i = 0; i < message.length; i++) {
  console.log(`${message[i]} ${message.charCodeAt(i).toString(16)}`);
// Output:
// h 68
// i 69
//   20
// � d83d
// � de0d

// iterate over Unicode code points
for (const char of message) {
  console.log(`${char} ${char.codePointAt(0).toString(16)}`);
// Output:
// h 68
// i 69
//   20
// 😍 1f60d

This shows a few things:

  1. h, i, and [space] correspond to single UTF-16 code units, \u0068, \u0069, and \u0020 respectively
  2. the 😍 emoji is comprised of a surrogate pair: \ud83d\ude0d; each value alone is invalid and prints as , but together they are interpreted correctly as the emoji
  3. the length of a string in JS corresponds to the number of UTF-16 code units, NOT Unicode code points, and NOT bytes
  4. when you .split('') a string, or access its indexes, it uses the UTF-16 code points
  5. on the other hand, when you iterate over a string with for...of, or anything that uses the iterable protocol, each loop yields the full unicode character, which could be multiple UTF-16 code units in length

We can also decode and encode UTF-8 in JS:

const bytes = new Uint8Array([
  0x68, // h
  0x69, // i
  0x20, // [space]
  0xf0, 0x9f, 0x98, 0x8d, // 😍

const string = new TextDecoder('utf-8').decode(bytes); // => "hi 😍"
string.length; // => 5

console.log(new TextEncoder().encode('😍')); // => Uint8Array(4) [ 240, 159, 152, 141 ]

Note that the emoji is represented by 4-bytes in UTF-8, rather than the surrogate pair in UTF-16, but in both encodings it would occupy 32 bits. The Latin characters have the same numeric values, but UTF-16 is using 16 bits per character, making UTF-8 more space-efficient.

CharacterCode PointUTF-8 HexUTF-8 BinaryUTF-16 HexUTF-16 Binary
hU+0068680110100000 6800000000 01101000
iU+0069690110100100 6900000000 01101001
[space]U+0020200010000000 2000000000 00100000
😍U+1F60DF0 9F 98 8D11110000 10011111 10011000 10001101D8 3D DE 0D11011000 00111101 11011110 00001101

Extended Grapheme Clusters

There is one more layer to this onion, and that is the “grapheme cluster”. Grapheme clusters are certain sequences of Unicode code points that should be displayed as one visual unit.

Common examples are modified emoji, like thumbs up with a skin tone: 👍🏽.

This symbol is actually comprised of two code points: Thumbs Up (U+1F44D) and a skin tone modifer (U+1F3FD). In UTF-16, each of those code points is comprised of a surrogate pair. In UTF-8, each would be a 4-byte value.

const grapheme = '👍🏽';
grapheme.length; // => 4
[...grapheme]; // => [ '👍', '🏽' ]
for (const char of grapheme) {
  console.log(`${char} ${char.codePointAt(0).toString(16)}`);
// Output:
//  👍 1f44d
// 🏽 1f3fd

In Summary

As a programmer, knowing how encodings work is useful when you are slicing, dicing, counting, splitting, or in any way manipulating or parsing strings. A mental model of Unicode and character encoding is prerequisite to understanding these operations.

As shown in the JS examples above, some functions operate on UTF-16 code units (String.length, indexes), while others operate on Unicode code points (iterables such as for...of loops and ... spread). In other languages string methods may operate on bytes.

Handling grapheme clusters is a layer above the encoding, and often times languages don’t have a built-in way to work with them. The JavaScript MDN docs simply concede:

Iterating through grapheme clusters will require some custom code.