Why the cut points decide what your RAG can answer
In a retrieval pipeline the chunk is the atom: it is what gets embedded, what gets scored against the query, and what gets pasted into the prompt. A chunk that is too large dilutes its own embedding — one vector has to represent several topics, so it matches everything weakly. A chunk that is too small loses the context that made it meaningful: "it costs 12,000 won" retrieves perfectly and answers nothing.
The failure that is hardest to debug is a fact split across a boundary. If the product name is at the end of chunk 3 and the price is at the start of chunk 4, no single chunk answers "how much is X" — both retrieve, neither is sufficient, and the model either says it does not know or invents the missing half. Overlap exists to make that less likely; seeing where the boundaries actually fall is how you find out whether it worked.
That is what this page shows. Instead of a chunk list you get the original document with every chunk drawn over it, so a boundary landing in the middle of a table row, a code block or a sentence is visible immediately — the kind of thing a printed list of chunks hides by making every chunk look equally tidy.
How RecursiveCharacterTextSplitter really splits
The splitter holds a separator list from coarse to fine — the default is ["\n\n", "\n", " ", ""] — and picks the first separator that occurs anywhere in the text. It cuts on that separator, then packs the resulting pieces back together, adding one piece at a time until the next one would push the chunk past chunk_size. Any piece that is still too big on its own is recursed into with the remaining, finer separators.
Two details decide most of the surprising output. A piece counts as small enough when its length is strictly less than chunk_size, not less than or equal. And when the separator list runs out and a piece is still oversized, it is emitted as-is — LangChain only logs a warning, and this page marks the chunk instead of hiding it.
A measured run makes it concrete. A 296-character English text at chunk_size 120, chunk_overlap 40 produces five chunks of 42, 1, 120, 117 and 54 characters. The one-character chunk is the newline left between two paragraphs — the splitter treats it as a piece like any other, and it is not a bug you introduced. Coverage is 296 of 296 characters with no gap.
The overlap you set is a ceiling, not a promise
Overlap is built by dropping whole pieces from the front of the previous chunk until what remains fits the overlap budget. Because pieces are indivisible, what is left over is whatever it happens to be — usually less than you asked for, and sometimes nothing. In the 296-character run above, chunk_overlap was set to 40 and the measured result was: maximum overlap 38, average 9.5, and three of the four boundaries sharing no characters at all.
The character splitter with an empty separator behaves completely differently, because there every character is its own piece and the arithmetic is exact. On 400 identical characters: size 100 with overlap 0 gives 4 chunks at stride 100; size 100 with overlap 80 gives 16 chunks at stride 20 with 5 chunks covering each point; size 10 with overlap 9 gives 391 chunks at stride 1. The depth is ⌈size/(size − overlap)⌉ exactly.
So the number to trust is the measured one. This page reports the average and maximum observed overlap next to the value you typed, and counts how many boundaries share nothing — that last figure is the one to watch, because those are exactly the places where a fact can fall through the crack.
Characters or tokens — and why Korean changes the answer
LangChain’s default length function counts characters (UTF-16 code units), so chunk_size = 1000 means a thousand characters. Embedding models, however, are limited in tokens. English prose runs about 5.11 characters per token and Korean about 1.68, measured on this page — so a 512-character chunk is roughly 100 tokens of English but 300 tokens of Korean, and the same setting fills three times as much of the model’s window.
Switching this page to token length runs the real BPE tokenizer and re-splits, so the boundaries move in front of you. Cutting is still done on character boundaries even in token mode — measuring in tokens but cutting on token indices would break Hangul syllables on screen, since 6–12% of o200k_base token boundaries fall inside a syllable.
Token length is expensive, so it is memoised per span and capped: after 50,000 tokenizer calls the splitter falls back to character length and says so, rather than freezing the page on a very long document. For a first pass, splitting by characters and reading the token counts afterwards is usually enough.
Separator presets: prose, Markdown, code, Korean
Every separator here is a literal string, never a regular expression — LangChain JS escapes separators before use, so a pattern like "\n#{1,6} " never worked there either, and reproducing LangChain exactly is the point of this page.
Default is paragraph, line, space, character. Markdown adds heading markers ("\n# " through "\n#### "), fenced code ("\n```") and horizontal rules before falling back to the default list, so sections stay whole. Code leads with class, def, function, export, const, let, if, for and while boundaries, which keeps a function body from being cut in half.
The Korean preset adds sentence endings — "다. ", "요. ", "까? ", "。" and the rest — and only makes sense together with keepSeparator: "end". With the separator kept at the start (the LangChain default) every chunk would begin with ". ", which is exactly why generic chunk visualisers are useless for Korean text.
- keepSeparator has three settings. "start" attaches the separator to the following piece, "end" to the preceding one, and "off" discards it — which is the only setting that leaves gaps in the coverage, because the discarded characters belong to no chunk. The coverage figure on this page makes that visible instead of leaving you to discover it in the embeddings.
Matching your embedding model — and the cost
Start from the model’s ceiling, not from a blog post. OpenAI text-embedding-3-small and -large accept 8,191 tokens but perform best around 256–512; Cohere embed-v3 hard-stops at 512 tokens while embed-v4 goes to 128,000; BAAI/bge-m3 takes 8,192; all-MiniLM-L6-v2 stops at 256 and silently truncates anything longer; the general BERT family stops at 512.
The four built-in size presets are starting points, not recommendations: 512/128 (the Microsoft guidance, 25% overlap), 1000/200 (the LangChain default), 256/32 for precise lookup, and 300/0 for code, where overlap duplicates function bodies for no benefit. Load one, look at the boundaries on your own document, and adjust from what you see.
Budget with the chunk total, not the document total. Overlapping text is embedded once per chunk, so the sum of chunk tokens exceeds the document’s token count — measured here, a 500-character chunk with 100 characters of overlap inflated the total by about 21%, while the same text at zero overlap inflated it not at all. Embedding APIs bill the sum, and so does the storage in your vector database.
Reproducibility traps, and getting the chunks out
LangChain JS and LangChain Python do not always agree. Their merge loops account for separator length differently, and on a 28-character fixture split at size 14 with overlap 7 the JS dialect produces four chunks while the Python one produces three. On the default recursive screen the merge separator is empty and both branches collapse to identical arithmetic — which is why the difference only bites in fixed-separator mode with the separator discarded.
Line endings are the other quiet source of drift. A document saved on Windows carries CRLF, and "\r\n" is two characters where "\n" is one, so every chunk boundary shifts and the token counts change. This page detects CRLF and normalises it, the same thing your ingestion script should do before it splits anything.
When the split looks right, export JSON or JSONL. Each record carries the chunk text, its id, its character span in the original document and its token count, so a retrieved chunk can be traced back to the exact offset it came from — which is what you need to build a citation, and what a list of bare strings cannot give you.
- Working limits: 200,000 characters of input, 2,000 chunks and 4,000 highlight segments drawn at once. Past those the page tells you it is showing a partial view rather than quietly rendering something wrong — for a book-length corpus, tune on a representative chapter and apply the settings in your pipeline.