How UTF-8 Works: Its Origins, Byte Length, and UTF-16 Compared

ASCII is a seven-bit encoding with 128 values, including letters, digits, punctuation, and control characters. It cannot represent the full range of written languages. Different encodings developed for different requirements: Shift_JIS in Japan, EUC-KR for Korean, Big5 for traditional Chinese, and ISO 8859 encodings, among others. Interpreting bytes under the wrong encoding produces garbled text.

Unicode provides a shared framework for code points and text processing. Its consortium was established in January 1991; the first volume of Unicode 1.0 followed in October. Assigning a number to a character and serializing that number into bytes are related but distinct jobs.

The early 16-bit model must be distinguished from variable-length UTF-16 with surrogate pairs. Serializing the value for A as two bytes gives 00 41 or 41 00, depending on byte order. A C routine that treats a zero byte as the end of a string cannot simply read such data as an ordinary byte string.

ASCII-based separators and file-name handling also mattered to Plan 9. A useful solution needed to accommodate existing byte-oriented software, avoid introducing ASCII delimiter bytes inside other characters, and keep plain ASCII compact. That does not mean every existing operation would automatically become Unicode-aware.

RFC 3629 distinguishes Thompson’s design work from Pike’s design criteria. The X/Open Joint Internationalization Group helped take the design through standardization. FSS-UTF, UTF-2, and UTF-8 were names used along the way.

Details of the often-retold diner story should not be confused with the historical account supported by the RFC. The engineering goal is the more useful guide here: keep ASCII-based syntax recognizable while allowing a much larger repertoire of text.

The x positions carry bits from the code point. ASCII retains its original byte values, and those values do not appear inside a multibyte sequence. Continuation bytes start with 10. Within a valid sequence, moving back at most three bytes locates its start; this is synchronization, not a promise to repair corrupted data.

The leading pattern alone is not enough to validate UTF-8. U+D800–U+DFFF are reserved for UTF-16 surrogates and cannot be encoded directly. C0 80 is an overlong encoding; ED A0 80 encodes a surrogate; F4 90 80 80 exceeds the upper limit. All are invalid, as are incomplete sequences.

UTF-8 has no byte-order choice. An initial EF BB BF may be used as a BOM signature, but it does not specify endianness. Whether it is expected or allowed depends on the consumer or protocol.

A zero byte represents U+0000. C string functions still perform byte-level operations: strlen does not count visible characters, and strcmp does not implement language-sensitive collation.

Many kana and CJK characters in the Basic Multilingual Plane use three UTF-8 bytes. Supplementary-plane characters use four. Combining sequences can use several code points for one visible unit, so neither “one Japanese character” nor “one traditional Chinese character” implies a fixed byte count.

A comparison with legacy encodings depends on which characters those encodings can represent. File-size differences, including differences after gzip or Brotli compression, must be measured with actual content and settings.

Agree on the encoding at both ends and define how malformed input is handled. Reading UTF-8 bytes as another encoding can still corrupt the text. Saving after replacement of invalid bytes can lose information, so retain the original while diagnosing the problem.

RFC 2279, published in January 1998, described sequences of up to six bytes. RFC 3629 replaced it in November 2003 and limited the encoding to four bytes and the range through U+10FFFF, excluding surrogate code points.

For valid sequences of Unicode scalar values, conversion among UTF-8, UTF-16, and UTF-32 can preserve the text without loss. The size of the code space is not the number of assigned characters: some positions are unassigned. Annual character counts or predictions about future capacity are not needed to understand the encoding.

Keep an HTML encoding declaration, an HTTP charset declaration where applicable, and the actual file bytes consistent. Changing a declaration does not convert the file. The binary editor can help you inspect those bytes.

A percentage of websites using an encoding is not a percentage of web traffic. Instead of relying on a percentage without a defined population or measurement date, check compatibility and the actual data you exchange.

These sizes exclude a BOM. Supplementary-plane characters use four bytes in both UTF-8 and UTF-16. Even UTF-32 does not make a visible character a fixed-size unit when combining sequences are involved.

Unicode 2.0 (1996) introduced surrogates and UTF-16; supplementary characters were first added in Unicode 3.1 (2001). The encoding mechanism and the assignment of characters are separate milestones. UTF-16 uses one or two 16-bit code units per scalar value. JavaScript’s String.length counts those units: "😀".length === 2. In contrast, new TextEncoder().encode("😀").length === 4 measures UTF-8 bytes. Choose the measurement that matches the task.

• UTF-8 preserves ASCII and encodes Unicode scalar values in one to four bytes.
• Separate the 1992 design history from the later RFC revisions.
• Overlong encodings, surrogates, out-of-range values, and incomplete sequences are not valid UTF-8.
• Distinguish bytes, code units, code points, and visible characters.
• Agree on charset, BOM handling, and error handling, then check the converted data.