Learn / One concept at a time
Tokens: the pieces AI reads
Two words can be two tokens. One word can be six. See what changes.
By the end, you can:
Explain why counting words cannot tell you exactly how many tokens a text model will use.
01 · Make a guess
A long word is still one word. Must it use just one token? Make your prediction before opening the second example.
Take a moment to predict it. You don’t need to know any jargon.
Reveal the explanation
No. In this encoding, antidisestablishmentarianism becomes six pieces. Word boundaries and token boundaries are different things.
See it happen
From your words to token IDs.
The same message becomes text pieces, then a sequence of IDs. The text is preserved through the transformation.
You typehello world
2 tokens · cl100k_base
Two tokens here. The second piece begins with a space; the display must keep it. The official test records these IDs, and the encoding table confirms their text.
Read the token-by-token version
"hello"→ token ID15339" world"→ token ID1917
You typeantidisestablishmentarianism
6 tokens · cl100k_base
One word, six tokens. These pieces and IDs are the documented result for this named encoding, not a live model response.
Read the token-by-token version
"ant"→ token ID519"idis"→ token ID85342"establish"→ token ID34500"ment"→ token ID479"arian"→ token ID8997"ism"→ token ID2191
Step 1 of 3 · Read the message
Exact recorded examples, not a live tokenizer. ␣ marks a space. Token IDs are labels, not scores. Other tokenizers can split the same text differently.
02 · The idea in plain English
Start with a tiny message
Before a text model processes your message, a tokenizer splits it into pieces and represents them with numbered IDs. In the hello world example, the space belongs to the second piece. Keep it: spaces are part of the text.
The number is a label
An ID identifies a piece in that encoding. A bigger ID does not mean a longer word or a more intelligent answer. Turning the complete sequence back into text recovers the original message.
The practical catch
Different encodings can split the same text differently. These stored examples use cl100k_base; they do not represent every model. A complete API request can also include message formatting and other inputs, so this text-only count is not a bill estimate.
Why this matters to you
Token counts help explain text limits and usage-based API costs. Count with the relevant tokenizer before estimating either; a word count is not an exact substitute.
03 · Use it once
Two prompts each contain 100 words. Can you promise they will use the same number of tokens? Why?
This note stays on this page and disappears when you leave. It isn’t sent or saved.
Compare your reasoning
No. Their spelling, spacing and language can change the pieces. The encoding also matters. Use the relevant tokenizer to compare the actual texts.
04 · Take one thing with you
Look away. Explain a token to someone who has never heard the term, then explain why one word can become several.
A sentence to remember
A token is a piece of text represented by an ID. A tokenizer can divide a word into smaller pieces, so words and tokens are not interchangeable counts.
That’s one new idea.
You’ve reached the end.
Practice is saved only in this browser. It isn’t a test score.
Back to the concept library →Follow the evidence
Where this lesson comes from.
Reviewed 25 September 2026. The named examples are reproducible fixtures, not claims about every AI model.
- Understanding and counting tokens ↗OpenAI Help Center · checked 24 September 2026What tokens are; effects of text, language and encoding; token-based limits and usage; additional tokens in complete API requests.
- How to count tokens with tiktoken ↗OpenAI Cookbook · checked 24 September 2026Exact six-piece cl100k_base long-word fixture and IDs; encoding/decoding; differences across encodings. This archived article is used for named-encoding examples, not current model mappings or prices.
- tiktoken: official encoding tests ↗OpenAI · checked 24 September 2026test_simple and test_basic_encode record hello world → [15339, 1917] for cl100k_base; test_simple also verifies decoding recovers the original text.
- cl100k_base encoding table ↗OpenAI · checked 24 September 2026Token 15339 decodes to hello; token 1917 decodes to a leading space followed by world. This is a tokenizer data file, not a reader-friendly article.
Found an error? Send a correction.