How BPE turns a sentence into tokens
Byte-pair encoding starts from raw UTF-8 bytes and repeatedly merges the most frequent adjacent pair until it has a vocabulary of fixed size. Encoding then replays those merges greedily, so the split you see is not a grammatical decision — it is whichever merges happened to be learned from the training corpus.
Spaces belong to the word that follows. “hello” and “ hello” are both a single token but different ids, and “ double space” costs four tokens because each extra space becomes a token of its own. Stray indentation in a prompt is therefore not free.
Case and punctuation change the split too: “hello” is one token but “HELLO” is two (“HEL” + “LO”), and JSON is expensive because quotes, colons and braces break every merge — {"user_id": 123, "name": "kim"} costs 13 tokens for 30 characters.
Reading the blocks on this page
One coloured block is one token; a dashed block with an ×n badge is a single character that cost n tokens, which BPE does whenever a character is rare enough to fall back to raw bytes. Hover any block to see its ids and its bytes.
Korean shows both extremes at once. 안녕하세요 is five characters but only two tokens (“안” + “녕하세요”), while 뷁 alone costs two and 쀍 costs three. Frequency, not length, decides.
Emoji behave the same way: 😀 is one token, but the family 👨👩👧👦 spans seven blocks and eleven tokens — four people at two tokens each plus three zero-width joiners. Numbers are grouped in threes, so 1234567 is “123” + “456” + “7”.
o200k_base vs cl100k_base
cl100k_base (GPT-4, GPT-3.5) holds about 100,000 entries; o200k_base (GPT-4o, GPT-5 and later) about 200,000. On English the extra vocabulary changes almost nothing: a 158-character English paragraph is 30 tokens under both.
Outside English the gap is large. The same Korean paragraph is 42 tokens under o200k_base and 73 under cl100k_base — 74% more. Even single words move: 한국어 is 2 tokens versus 4, and 안녕하세요 is 2 versus 5.
Practical rule: benchmark non-English prompts on the encoding your production model actually uses. A prompt that comfortably fits a context window under o200k_base may overflow it on an older cl100k_base model, and the cost estimate will be wrong in the same direction.
Special tokens and chat scaffolding
Chat models wrap messages in reserved tokens such as <|im_start|>, <|im_sep|> and <|im_end|>. They are single tokens with ids outside the text range — <|im_start|> is 200003 in o200k_base and 100264 in cl100k_base — and this page probes the encoder for those ids rather than hard-coding them.
The “read special tokens” switch matters for security, not just for counting. With it off, a pasted <|im_end|> is encoded as ordinary characters — which is exactly what happens to user input in a real API call, and why the delimiter cannot be used to escape a system prompt.
The scaffolding is not free: a two-message conversation carries several of these tokens plus role names, which is the fixed overhead every chat request pays before any of your own text is counted.
Why token boundaries explain odd model behaviour
The classic “how many r’s in strawberry” failure is a tokenisation artefact: the model never sees ten letters, it sees three tokens — “st”, “raw”, “berry”. Character-level tasks (counting letters, reversing a string, rhyming, acrostics) are hard for the same reason.
It also explains formatting quirks. A prompt ending in a trailing space puts the model in the middle of a merge and often degrades the completion; stop sequences must line up with real token boundaries; and identifiers split oddly — getUserName() becomes “ get” + “User” + “Name” + “();”.
And it explains cost asymmetry between languages. A 41-character URL costs 14 tokens; the same length of English prose costs about 9. Dense punctuation, base64 blobs and minified JSON are the worst offenders in a prompt budget.
Common surprises and how to check them
- A tab is its own token, and CRLF line endings tokenise differently from LF. Normalising line endings before you build a prompt template removes a whole class of invisible drift.
- Sum of parts ≠ whole. Encoding two strings separately and adding the counts overshoots, because a merge that would have crossed the join never happens. Always tokenise the final assembled prompt.
- Token ids are per-encoding, not universal. The same id means different text in o200k_base and cl100k_base, so never store raw ids without recording which encoding produced them.
- Claude and Gemini boundaries cannot be shown honestly, because neither provider publishes an offline tokenizer. Use this page for the mechanism and the GPT numbers, and the token counter for estimated counts on those families.
What people use this for
Trimming prompts: paste the system prompt and look for blocks that buy nothing — repeated whitespace, decorative separators, verbose JSON keys. Shortening a key from "customer_email_address" to "email" removes tokens from every single request.
Preparing datasets: fine-tuning and embedding pipelines have hard token limits per record, and this page shows where a record will be truncated instead of leaving you to discover it in an API error.
Teaching and debugging: showing a colleague that 안녕하세요 is two tokens while 쀍 is three usually settles a design argument faster than a paragraph of explanation.