LLM Tokenizer Visualizer
Paste text and see every token as a coloured block — token IDs, whitespace, chat special tokens, and which characters cost the most. Runs in your browser.
One block = one token. A dashed block is a single character that cost more than one token.
Your text stays in this browser and never goes in the URL.
Off, <|im_start|> is seven ordinary characters. On, it becomes the single reserved token a chat API sends.
Token blocks
Paste text and the token blocks appear here.
Claude and Gemini
Anthropic and Google publish no tokenizer, so their token boundaries cannot be drawn honestly. Only the OpenAI encodings can be shown exactly.
Estimate Claude and Gemini token counts and API cost → LLM Token Counter
How to use
- Paste your text, or press one of the sample buttons.
- Pick an encoding — o200k_base for GPT-4o and GPT-5, cl100k_base for GPT-4 and 3.5.
- Hover a block to see its token IDs and bytes. Toggle “show whitespace” or switch to the token ID list.
- Check the chart below to see which character types consume the most tokens.
FAQ
Why don’t token boundaries line up with words?
BPE absorbs a leading space into the following word. “Hello world” is two tokens, “Hello” and “ world” — the space belongs to the second one. That is also why “Hello” and “ Hello” are different token IDs, and why a stray double space quietly costs you an extra token.
Why does one Korean character become several tokens?
o200k_base holds common syllables whole but falls back to raw UTF-8 bytes for rare ones. Measured on this page: 뷁 costs 2 tokens and 쀍 costs 3. Those places are drawn with a dashed border and an ×n badge, and the inspector shows the actual bytes.
Why do emoji cost so many tokens?
😀 is a single token, but the family 👨👩👧👦 costs 11 tokens across 7 blocks: each of the four people is 2 tokens, and each of the three zero-width joiners holding them together is a token of its own (4 × 2 + 3 = 11). One grapheme legitimately spanning several blocks is a fact of BPE, not a bug in the display.
What is the difference between o200k_base and cl100k_base?
o200k_base has a roughly twice larger vocabulary and packs non-English text far better. The same Korean paragraph typically needs about 45–50% more tokens under cl100k_base. o200k_base is what GPT-4o and later use; cl100k_base is GPT-4 and GPT-3.5. Use “Compare encodings” on this page to measure your own text.
What are special tokens like <|im_start|>?
They are reserved tokens a chat API uses to mark roles and message boundaries. Even a two-message conversation carries several of them as scaffolding. Turn on “read special tokens” and they appear as purple blocks; leave it off and they are counted as ordinary characters, which is what happens when a user pastes that text into your prompt.
Why can’t you show Claude or Gemini token boundaries?
Anthropic and Google publish no tokenizer you can run offline. Drawing OpenAI boundaries and relabelling them “Claude”, as some sites do, is simply false. For an estimated token count and API cost for those models, use the LLM Token Counter on this site.
Related tools
More AI & Data tools: AI & Data