RAG Text Chunking & Overlap Visualizer
See exactly where your text is split and where chunks really overlap. Recursive and character splitters with real BPE token counts, in your browser.
Recursive splitting reproduces LangChain exactly — the overlap you set is a ceiling, not a promise. Nothing leaves your browser.
Text
Splitter
Walks the separator list from coarse to fine, exactly like LangChain’s RecursiveCharacterTextSplitter.
12.5% of chunk_size
\n\n · \n · \s · \e
Advanced
Statistics
In this mode the overlap you set is a ceiling. Real overlap depends on where the pieces fall and is often smaller — sometimes zero. LangChain measures length in UTF-16 code units, so one emoji counts as two or more. This page counts the same way.
Paste a document, or start from one of the samples.
Recommended chunk size by embedding model
| Model | Max input tokens | Suggested chunk |
|---|---|---|
| OpenAI text-embedding-3-small | 8,191 | 256–512 · A large limit is not a reason to use large chunks. |
| OpenAI text-embedding-3-large | 8,191 | 256–512 |
| OpenAI text-embedding-ada-002 | 8,191 | 256–512 |
| Cohere embed-v3 | 512 | 256–450 · Low limit — the over-limit warning genuinely matters here. |
| Cohere embed-v4 | 128,000 | 512–1,024 |
| BAAI/bge-m3 | 8,192 | 512–1,024 |
| Nomic embed-text-v2 | 8,192 | 512–512 |
| all-MiniLM-L6-v2 | 256 | 128–200 · Anything past the limit is silently truncated. |
| BERT-family (general) | 512 | 256–400 |
Sources disagree. Microsoft suggests 512 tokens (≈2,000 characters) with 25% overlap; the common community figure is 250–500 tokens with 10–20% overlap; other reports found 200 tokens with no overlap best for RecursiveCharacterTextSplitter. Checking against your own documents is what this tool is for.
How to use
- Paste or open a document — the split appears immediately.
- Set chunk size and overlap, or start from a preset.
- Hover a chunk to see its range, characters and tokens; overlapping text is marked separately.
- Export the chunks as JSON or JSONL for your embedding pipeline.
FAQ
What chunk size should I use?
The published advice disagrees with itself. Microsoft suggests 512 tokens with 25% overlap, the common community figure is 250–500 tokens with 10–20% overlap, and other reports found 200 tokens with no overlap worked best with RecursiveCharacterTextSplitter. Because the right answer depends on your documents and your retriever, this page ships those four starting points as presets and then lets you see what each one actually does to your text.
How much overlap should I set?
Ten to twenty-five percent is the usual range. The more important point is that in recursive splitting the number you set is a ceiling, not a guarantee: the splitter builds the overlap by dropping whole pieces from the front of the previous chunk, so the real overlap is whatever is left — often smaller and sometimes zero. That is why this page reports the measured average, the maximum, and how many boundaries share nothing at all.
What is the difference between character length and token length?
Character length counts UTF-16 code units, exactly like LangChain’s default. Token length runs the real BPE tokenizer. English prose averages four to five characters per token while Korean averages under two (measured here: 5.11 versus 1.68), so the same chunk_size of 512 holds about three times as much English as Korean. Switch between the two modes to see the boundaries move.
Why does Korean cost more tokens?
CJK text costs roughly one token per character, and o200k_base often splits a single Hangul syllable across two tokens — in our measurements 6–12% of token boundaries fall inside a syllable. That is also why a splitter that cuts on token indices breaks Korean characters on screen, and why this page measures token length but always cuts on character boundaries.
How does RecursiveCharacterTextSplitter actually split?
It walks the separator list from coarse to fine and cuts on the first separator that appears in the text. The resulting pieces are packed back together until adding one more would exceed chunk_size, at which point the chunk is closed. The overlap is then built by dropping pieces from the front until what remains fits the overlap budget. Everything happens at piece granularity, which is why the result rarely matches the numbers you typed.
Why is the sum of the chunk tokens larger than the whole text?
Two reasons: the overlapping stretches are counted once per chunk, and characters that merged into a single BPE token in the middle of the text get split at a chunk edge. Overlap dominates — in our measurements a 500-character chunk with 100 characters of overlap inflated the total by about 21%, while the same text with no overlap inflated it by nothing at all. Embedding APIs bill the chunk sum, so budget with that number.
Related tools
More AI & Data tools: AI & Data