Character Data Types

Imagine you're building a communication system in 1963. Your users are American engineers who need to exchange messages containing English letters, digits, punctuation, and some control codes for their teletypes. You decide: 7 bits per character gives you 128 possible values—more than enough.

This decision worked brilliantly for decades. And then the world got connected.

Suddenly, your Japanese colleague wants to send you a message with 日本語 characters. Your French partner needs to include café with its accent. Your Greek customer writes in Ελληνικά. And none of these fit in your 128-character system.

This is the story of ASCII and Unicode—a tale of a simple solution that worked until it didn't, and the elegant (if complex) solution that rescued global computing.

ASCII (American Standard Code for Information Interchange) was developed in 1963 and became the dominant character encoding for early computing. Understanding its design illuminates both its brilliance and its limitations.

The 7-bit constraint:

ASCII uses 7 bits per character, providing 2⁷ = 128 possible values (0-127). This wasn't arbitrary—7 bits was practical for the communication equipment of the era, and 128 values seemed sufficient for the anticipated use cases.

The allocation of 128 codes:

The ASCII designers carefully allocated these precious 128 values:

Clever design decisions:

The ASCII designers made several elegant choices that survive in modern computing:

1. Digits are sequential: '0'-'9' occupy codes 48-57, enabling digit_value = char_code - 48

2. Letters are sequential: 'A'-'Z' (65-90) and 'a'-'z' (97-122) enable easy alphabetic calculations

3. Case bit: Uppercase and lowercase differ by exactly 32 (bit 5). 'A' (65) + 32 = 'a' (97). This enables fast case conversion via a single bit flip.

4. Collation-friendly: Letters and digits sort correctly in code order

5. Control characters first: Codes 0-31 are non-printable controls, clearly separated from printable characters

ASCII was designed by Americans, for Americans, in an era when computing was primarily an English-language activity. As computing globalized, its limitations became painfully apparent.

Problem 1: No accented characters

European languages need characters ASCII lacks:

• French: é, è, ê, ë, ç, à, ù, ô
• German: ä, ö, ü, ß
• Spanish: ñ, ¿, ¡
• Swedish: å, ä, ö
• Polish: ą, ć, ę, ł, ń, ó, ś, ź, ż

ASCII has no room for these. European computer users faced a choice: abandon their language's proper spelling, or create incompatible extensions.

Problem 2: Non-Latin scripts are impossible

For billions of users, ASCII doesn't even contain their alphabet:

The 8-bit extension chaos:

Computer manufacturers tried to solve this by using the 8th bit (extending 128 codes to 256). But different vendors assigned different characters to codes 128-255:

• ISO-8859-1 (Latin-1): Western European languages
• ISO-8859-2: Central European languages
• ISO-8859-5: Cyrillic
• Windows-1252: Microsoft's Western European variant
• Windows-1251: Microsoft's Cyrillic variant
• KOI8-R: Soviet-era Russian encoding

The result was encoding chaos: a file saved in one encoding displayed as garbage in another.

Problem 3: Asian languages need thousands

The 256-character extensions still couldn't handle Chinese, Japanese, or Korean. These languages use logographic or syllabic scripts with thousands of characters:

• Chinese: ~50,000+ characters (though ~3,000 for daily use)
• Japanese: ~2,000+ kanji plus hiragana and katakana
• Korean: ~11,000+ hangul syllable blocks

Various multi-byte encodings were developed (Big5, GB2312, Shift_JIS, EUC-KR), each incompatible with the others. A Japanese website couldn't reliably display Chinese characters, and vice versa.

The core problem:

Every country or region developed its own encoding. There was no single encoding that could represent text from multiple languages in the same document. A truly international web, email system, or software application was impossible.

In the late 1980s, engineers at Xerox and Apple began developing a universal character set that would solve the encoding chaos once and for all. Their solution became Unicode.

The Unicode philosophy:

Unicode is built on a simple but revolutionary principle: assign a unique number to every character in every writing system that has ever existed (and some that are being created).

Unlike ASCII's 128 characters or the 8-bit extensions' 256, Unicode provides space for over 1.1 million code points. This is enough for:

• All modern languages in active use
• Historical scripts (Egyptian hieroglyphs, Cuneiform, Mayan)
• Mathematical and technical symbols
• Musical notation
• Emoji (yes, 💩 has a Unicode code point)
• Private use areas for custom applications
• Future scripts not yet encoded

The code point concept:

Every character in Unicode is assigned a code point—a unique number in the range 0 to 1,114,111 (0x10FFFF in hexadecimal). Code points are written with the prefix U+ followed by the hexadecimal value:

• U+0041 = 'A' (Latin capital letter A)
• U+03B1 = 'α' (Greek lowercase alpha)
• U+4E2D = '中' (Chinese character for 'middle')
• U+1F600 = '😀' (Grinning face emoji)

With over a million possible code points, Unicode needed an organizational structure. The entire range is divided into planes, each containing 65,536 (2¹⁶) code points.

The 17 Unicode planes:

Unicode defines 17 planes, numbered 0-16:

The Basic Multilingual Plane (BMP):

The BMP (Plane 0) is the most important. It contains:

• All ASCII characters (U+0000–U+007F)
• Latin Extended characters for European languages
• Greek, Cyrillic, Hebrew, Arabic scripts
• Devanagari, Bengali, Thai, and other Indic scripts
• Chinese, Japanese, Korean characters for everyday use
• Common punctuation and symbols
• The vast majority of text you'll encounter

Beyond the BMP:

Characters outside the BMP (code points ≥ U+10000) include:

• Historical scripts: Egyptian hieroglyphs, Cuneiform, Linear A/B
• Rare CJK characters used in specialized contexts
• Musical symbols and notations
• Mathematical alphanumeric symbols
• Emoji: Most emoji are in the SMP (Plane 1)
• Game symbols: Dominos, playing cards, Mahjong tiles

Unicode blocks:

Within each plane, characters are organized into contiguous blocks—ranges of code points assigned to specific scripts or purposes:

• Basic Latin (U+0000–U+007F): ASCII characters
• Latin-1 Supplement (U+0080–U+00FF): Western European extensions
• Greek and Coptic (U+0370–U+03FF): Greek alphabet
• Cyrillic (U+0400–U+04FF): Russian, Ukrainian, etc.
• Arabic (U+0600–U+06FF): Arabic script
• CJK Unified Ideographs (U+4E00–U+9FFF): ~20,000 common CJK characters
• Emoticons (U+1F600–U+1F64F): Face emoji

This block organization helps font designers, keyboard developers, and software engineers work with specific scripts.

A critical distinction that causes endless confusion: characters, code points, and bytes are three different things.

Character:

A character is an abstract concept—a unit of written language. The letter 'A', the digit '7', the emoji '😀' are all characters. Characters exist in human writing systems; they're the meaning we want to represent.

Code Point:

A code point is Unicode's numeric assignment to a character. It's the bridge between the abstract character and its representation. The character 'A' has code point U+0041 (decimal 65). The emoji '😀' has code point U+1F600 (decimal 128512).

Byte(s):

Bytes are how code points are actually stored in computer memory or transmitted over networks. The same code point can be represented by different byte sequences depending on the encoding used:

Why multiple encodings?

Unicode defines the what (which code points exist), but not the how (how to store them). Different encodings optimize for different use cases:

• UTF-8: Compact for ASCII-heavy text, variable-width, ASCII-compatible
• UTF-16: Efficient for CJK-heavy text, fixed-width for BMP, used by Java/JavaScript/Windows
• UTF-32: Simple (always 4 bytes), but wasteful; rarely used for storage

The choice of encoding affects:

• Storage size
• Processing speed
• Compatibility with legacy systems
• Transmission efficiency

Among Unicode encodings, UTF-8 has emerged as the dominant choice for the web, files, and modern systems. Understanding why illuminates important encoding principles.

UTF-8's variable-width design:

UTF-8 uses 1 to 4 bytes per character, depending on the code point:

The clever byte patterns:

UTF-8's byte patterns are engineered for safety and self-synchronization:

• 1-byte characters (ASCII): 0xxxxxxx — starts with 0
• 2-byte characters: 110xxxxx 10xxxxxx — lead byte starts with 110
• 3-byte characters: 1110xxxx 10xxxxxx 10xxxxxx — lead byte starts with 1110
• 4-byte characters: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx — lead byte starts with 11110

Continuation bytes always start with 10. This means:

• You can tell lead bytes from continuation bytes instantly
• If you start reading mid-character, you can scan for the next lead byte
• Invalid byte sequences are detectable

Let's crystallize the key differences between ASCII and Unicode:

We've traced the evolution from ASCII's 128 characters to Unicode's universal character set. Let's consolidate the key insights:

What's next:

Understanding that Unicode exists is step one. The next page explores why Unicode matters in modern systems—the practical implications for software engineering, the bugs that arise from ignoring Unicode, and why proper character handling is a non-negotiable skill for professional developers.