What Is a Token in an LLM? Tokens vs Words, Explained

The model never reads your words. It reads tokens — chunks of text produced by a tokenizer that runs before the model does anything. That one fact explains your context window, your API bill and your rate limits, and it is the whole of Day 1 of LLM Concepts.

The model never reads your words

Whatever you put in a prompt — a question, a pasted file, a JSON blob, a code block — is not what the model receives. The text is cut into chunks first. Those chunks are tokens, and everything downstream counts them: not characters, not words.

That distinction is invisible from the outside. You paste text, you get an answer, and nothing in the interface tells you how much text the model actually saw. So it is easy to reason in words and be quietly wrong.

It starts to matter the moment you try to estimate something. A 200-word prompt containing a table of dates is not the same size as 200 words of plain prose, because the split is different. Your editor shows one number, the pipeline bills another, and the gap between them is where surprise invoices and "why is this slow" questions come from.

The useful mental model for everything below is simple: text goes in, chunks come out, and every limit you meet from here is measured in those chunks. If you have wired up even a small LLM chain, you have been paying in this currency without naming it — your first LLM chain is a good place to watch the same unit show up.

What a tokenizer actually does

A tokenizer cuts your text into chunks before the model runs. The obvious design would be one token per word, and that is not what happens.

Why not words? There are too many. A word-level vocabulary would have to cover every name, typo, product code and language variant, and it would still break the moment something new appeared.

The other extreme has the opposite problem: one token per character keeps the vocabulary tiny, but every word becomes a long row of pieces and the model has far more units to track.

So tokenizers work at a level between characters and words. Common words come out whole. Rare ones get chopped:

unbelievable → un-believ-able

That is the trade: a vocabulary small enough to manage, and long or unusual strings split into several tokens instead of one. The model never sees "unbelievable" as a single object; it sees pieces that reassemble into a pattern it recognises.

Roughly four characters per token in English

The working rule for English text: about four characters make one token. Flip it around and a thousand tokens is roughly 750 words.

That ratio is the one to keep in your head, because it turns the numbers in a dashboard into something you can estimate up front. A 750-word document is about 1,000 tokens. A 2,000-word article is somewhere around 2,700 tokens. You do not need an exact count to know whether something fits inside a context window.

It also gives you a fast sanity check on the numbers other people quote. When someone mentions a large context window, one division tells you roughly how much English prose that buys, and whether the document you want to hand over has any chance of fitting.

Not all text is English prose, and the rule gets worse elsewhere:

• Code and long numbers split into more pieces than plain prose.
• A date is not one token.
• Non-English text usually costs more tokens for the same meaning.

If you are budgeting a prompt that carries a code block, a table of dates or a non-English conversation, assume more tokens than a plain word count suggests. The word count you see in your editor is the wrong metric. This is also why prompt shaping pays off: swap a wall of prose for a schema-driven prompt and the same information arrives in fewer chunks.

Two questions from the video, since these are exactly the estimates the rule exists for.

Quiz one: which costs the most tokens — 'hello', a date, or 'the cat'? 'hello' is a common word. 'the cat' is two common words. A date is a string of digits that splits into more pieces than either, so that is the one.

Quiz two: roughly how many tokens are in a hundred English words? At roughly four characters per token, and with 1,000 tokens at about 750 words, a hundred words lands around 130 tokens. Pick your letter and drop it in the comments.

Where tokens actually hit you

Three places, all counting the same unit:

1. API pricing counts input tokens and output tokens. Both sides of the call are billed in chunks.
2. Your context window is a token budget, not a word count. The limit is how many chunks fit, not how many words.
3. Rate limits are measured in tokens per minute.

Output counts too. A long answer costs more than a short question, and because rate limits apply per minute, a batch job firing many large prompts will hit them faster than a chat session.

Once tokens are the unit, prompt size stops being a vague feeling and becomes a number you can estimate before you ship. That changes the question you ask. Not "is this prompt too long?" but "how many tokens per call, times how many calls per minute?" — the same arithmetic that decides whether a batch runs comfortably or falls over, which is what production deployment is mostly about.

Quick recap

• A token is a chunk of text, not a word.
• A tokenizer cuts text into chunks before the model runs.
• Common words come out whole; rare ones get chopped.
• English: roughly four characters per token; 1,000 tokens ≈ 750 words.
• Code, long numbers and dates split into more pieces.
• Non-English text usually costs more tokens for the same meaning.
• API pricing counts input and output tokens.
• The context window is a token budget; rate limits are tokens per minute.

If you remember one line from Day 1, make it this: count tokens, not words. The window, the bill and the rate limit are the same number wearing three different labels — estimate it once, and all three stop being surprises.