A token is the unit of text a language model reads and writes: usually a whole word, part of a word, a number, or a punctuation mark. Models split all input into tokens before processing it and generate replies one token at a time, so tokens are also the unit in which AI usage is priced, limited, and measured.
For a buyer, tokens matter more than they seem. The price of every model call, the amount of text a model can consider at once, and the time it takes to answer are all counted in tokens. The same message costs a different number of tokens depending on the language it is written in and the model that reads it, which makes "price per million tokens" a poor basis for comparing vendors on its own.
One customer message, split into tokens
Here is the same short support message in English and in Polish, split by a real tokenizer. Each shaded block is one token. A leading space belongs to the token after it.
9 words, 11 tokens
9 words, 19 tokens
Split with OpenAI's tiktoken library using the o200k_base encoding, which GPT-4o and later OpenAI models use.3 Other vendors' tokenizers split the same text differently. Counts apply to these two sentences only.
Common English words map to single tokens. Polish words, with their endings and diacritics, break into several pieces. The Polish message carries the same meaning in 73% more tokens, and on the older cl100k_base encoding the gap was larger: 12 tokens against 24.
How text becomes tokens
A model cannot read letters. A tokenizer turns text into a list of numbers, each standing for a piece of text in a fixed vocabulary, and turns the model's output numbers back into text. That process is tokenization.
Most tokenizers use byte-pair encoding or a close relative. The method starts from single characters or bytes and repeatedly merges the most frequent neighboring pairs into new vocabulary entries. Frequent words end up as one token; rare words, names, and codes are spelled out in several subword pieces.1 SentencePiece applies the same idea directly to raw text in any language.2
Because the vocabulary is learned mostly from English-heavy data, English gets the most efficient encoding. Languages with richer word forms, such as Polish, Czech, or Hungarian, usually need more tokens for the same meaning.4
Why tokens decide the bill
Model APIs charge per million tokens, with separate prices for input and output. On Anthropic's list prices as of November 2026, output tokens cost five times as much as input tokens for every current Claude model.5 A conversation's cost therefore depends on how much context the system sends and how long the replies are.
- Input: 3,000 tokens (instructions, history, retrieved policy) × $2 per million
- $0.006
- Output: 300 tokens × $10 per million
- $0.003
- One call
- $0.009
Worked example. The token counts are illustrative; the prices are Anthropic's list prices for Claude Sonnet 5.5.5 Prices change often, so confirm them with the vendor. An agent makes several calls per conversation, so multiply by the number of steps.
Three consequences follow. First, the instructions and documents sent with every message often cost more than the reply. Second, a Polish or Czech deployment can cost noticeably more per conversation than the same agent in English. Third, compare vendors on cost per resolved conversation measured on your own transcripts, not on the price per token.
Token and related units
| Term | What it is | Where it matters |
|---|---|---|
| Token | The unit a model reads and writes | Pricing, context limits, rate limits, speed |
| Word | A unit people count | Briefs and budgets; convert to tokens per language before estimating cost |
| Character | A single letter, digit, or symbol | Message length limits in channels such as SMS |
| Tokenization | The process of splitting text into tokens | Explains why languages and codes differ in cost |
| Embedding | A list of numbers that represents the meaning of a text | Search and retrieval; computed from tokens |
Where token counts change decisions
Budget per conversation
Measure input and output tokens per resolved conversation on real transcripts, then model cost at peak-season volume.
sample → count tokens → × volumePrice each language
An agent serving Poland, Czechia, and Germany has a different token cost per market. Measure each language, not only English.
language → tokens per message → costFit the documents
Long contracts and policy manuals may not fit in one request. Retrieve the relevant sections instead of sending everything.
chunk → retrieve → fit windowKeep replies short
Every output token adds generation time before the caller hears an answer. Short replies keep the call natural.
fewer tokens → faster replyToken mistakes that cost money
Estimating in words
ControlWord counts understate tokens, especially outside English. Count with the target model's own tokenizer or token-counting endpoint.6
Bloated prompts
ControlLong instructions and full histories are resent with every call. Trim them, summarize old turns, and use prompt caching where the vendor offers it.
Truncated replies
ControlAn output limit set too low cuts answers off mid-sentence. Set it per task and monitor how often it is hit.
Context overflow
ControlWhen a conversation exceeds the window, early turns are dropped or the call fails. Plan summarization or retrieval before it happens.7
Common questions
How many words is 1,000 tokens?
In English, usually somewhat fewer than 1,000 words, since common words are single tokens and longer ones split. In Polish and other inflected languages, fewer words fit in the same number of tokens. Count with the model's own tokenizer for anything that affects a budget.
Why do AI models use tokens instead of words?
A fixed vocabulary of subword pieces can represent any text, including new words, names, typos, and order numbers, while keeping the vocabulary small enough for the model to learn.
Are input and output tokens priced the same?
Usually not. Output tokens cost more because the model generates them one at a time. On Anthropic's price list the ratio is five to one.
Do different models count tokens the same way?
No. Each vendor, and sometimes each model generation, uses its own tokenizer, so the same text yields different counts. Compare costs on the same sample text.
References
- Sennrich, R., Haddow, B. and Birch, A. Neural Machine Translation of Rare Words with Subword Units. ACL 2016.
- Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018.
- OpenAI. tiktoken. Open-source BPE tokenizer and encodings for OpenAI models.
- Petrov, A. et al. Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023.
- Anthropic. Pricing. Claude documentation.
- Anthropic. Token counting. Claude documentation.
- Anthropic. Context windows. Claude documentation.