AI Glossary Models and how they work

Token (in AI)

  • Tidio Agents team
  • Updated November 19, 2026
  • 5 min read
  • 7 sources
Definition

A token is the unit of text a language model reads and writes: usually a whole word, part of a word, a number, or a punctuation mark. Models split all input into tokens before processing it and generate replies one token at a time, so tokens are also the unit in which AI usage is priced, limited, and measured.

For a buyer, tokens matter more than they seem. The price of every model call, the amount of text a model can consider at once, and the time it takes to answer are all counted in tokens. The same message costs a different number of tokens depending on the language it is written in and the model that reads it, which makes "price per million tokens" a poor basis for comparing vendors on its own.

Example

One customer message, split into tokens

Here is the same short support message in English and in Polish, split by a real tokenizer. Each shaded block is one token. A leading space belongs to the token after it.

English
My parcel hasn't arrived. Can I get a refund?

9 words, 11 tokens

Polish
Moja paczka nie dotarła. Czy mogę dostać zwrot pieniędzy?

9 words, 19 tokens

Split with OpenAI's tiktoken library using the o200k_base encoding, which GPT-4o and later OpenAI models use.3 Other vendors' tokenizers split the same text differently. Counts apply to these two sentences only.

Common English words map to single tokens. Polish words, with their endings and diacritics, break into several pieces. The Polish message carries the same meaning in 73% more tokens, and on the older cl100k_base encoding the gap was larger: 12 tokens against 24.

How it works

How text becomes tokens

A model cannot read letters. A tokenizer turns text into a list of numbers, each standing for a piece of text in a fixed vocabulary, and turns the model's output numbers back into text. That process is tokenization.

TextTokenizerToken IDsModelOutput IDsReply
Every model call starts and ends with the tokenizer. Input and output are counted separately.

Most tokenizers use byte-pair encoding or a close relative. The method starts from single characters or bytes and repeatedly merges the most frequent neighboring pairs into new vocabulary entries. Frequent words end up as one token; rare words, names, and codes are spelled out in several subword pieces.1 SentencePiece applies the same idea directly to raw text in any language.2

Because the vocabulary is learned mostly from English-heavy data, English gets the most efficient encoding. Languages with richer word forms, such as Polish, Czech, or Hungarian, usually need more tokens for the same meaning.4

Cost

Why tokens decide the bill

Model APIs charge per million tokens, with separate prices for input and output. On Anthropic's list prices as of November 2026, output tokens cost five times as much as input tokens for every current Claude model.5 A conversation's cost therefore depends on how much context the system sends and how long the replies are.

Cost of one model call (Anthropic list prices, November 2026)
cost = input tokens × input price + output tokens × output price
Input: 3,000 tokens (instructions, history, retrieved policy) × $2 per million
$0.006
Output: 300 tokens × $10 per million
$0.003
One call
$0.009

Worked example. The token counts are illustrative; the prices are Anthropic's list prices for Claude Sonnet 5.5.5 Prices change often, so confirm them with the vendor. An agent makes several calls per conversation, so multiply by the number of steps.

Three consequences follow. First, the instructions and documents sent with every message often cost more than the reply. Second, a Polish or Czech deployment can cost noticeably more per conversation than the same agent in English. Third, compare vendors on cost per resolved conversation measured on your own transcripts, not on the price per token.

Compared

Token and related units

TermWhat it isWhere it matters
TokenThe unit a model reads and writesPricing, context limits, rate limits, speed
WordA unit people countBriefs and budgets; convert to tokens per language before estimating cost
CharacterA single letter, digit, or symbolMessage length limits in channels such as SMS
TokenizationThe process of splitting text into tokensExplains why languages and codes differ in cost
EmbeddingA list of numbers that represents the meaning of a textSearch and retrieval; computed from tokens
In practice

Where token counts change decisions

Customer support

Budget per conversation

Measure input and output tokens per resolved conversation on real transcripts, then model cost at peak-season volume.

sample → count tokens → × volume
Multilingual service

Price each language

An agent serving Poland, Czechia, and Germany has a different token cost per market. Measure each language, not only English.

language → tokens per message → cost
Internal knowledge

Fit the documents

Long contracts and policy manuals may not fit in one request. Retrieve the relevant sections instead of sending everything.

chunk → retrieve → fit window
Voice agents

Keep replies short

Every output token adds generation time before the caller hears an answer. Short replies keep the call natural.

fewer tokens → faster reply
Risks

Token mistakes that cost money

Estimating in words

ControlWord counts understate tokens, especially outside English. Count with the target model's own tokenizer or token-counting endpoint.6

Bloated prompts

ControlLong instructions and full histories are resent with every call. Trim them, summarize old turns, and use prompt caching where the vendor offers it.

Truncated replies

ControlAn output limit set too low cuts answers off mid-sentence. Set it per task and monitor how often it is hit.

Context overflow

ControlWhen a conversation exceeds the window, early turns are dropped or the call fails. Plan summarization or retrieval before it happens.7

FAQ

Common questions

How many words is 1,000 tokens?

In English, usually somewhat fewer than 1,000 words, since common words are single tokens and longer ones split. In Polish and other inflected languages, fewer words fit in the same number of tokens. Count with the model's own tokenizer for anything that affects a budget.

Why do AI models use tokens instead of words?

A fixed vocabulary of subword pieces can represent any text, including new words, names, typos, and order numbers, while keeping the vocabulary small enough for the model to learn.

Are input and output tokens priced the same?

Usually not. Output tokens cost more because the model generates them one at a time. On Anthropic's price list the ratio is five to one.

Do different models count tokens the same way?

No. Each vendor, and sometimes each model generation, uses its own tokenizer, so the same text yields different counts. Compare costs on the same sample text.

Sources

References

  1. Sennrich, R., Haddow, B. and Birch, A. Neural Machine Translation of Rare Words with Subword Units. ACL 2016.
  2. Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018.
  3. OpenAI. tiktoken. Open-source BPE tokenizer and encodings for OpenAI models.
  4. Petrov, A. et al. Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023.
  5. Anthropic. Pricing. Claude documentation.
  6. Anthropic. Token counting. Claude documentation.
  7. Anthropic. Context windows. Claude documentation.

Tidio Agents

Build an agent that resolves, not only answers

We design and build custom AI agents for customer service, shopping and back-office work, on your infrastructure, with the code handed over to you. Projects start with a 21-day private AI audit.

Request a call