What Are Tokens in AI?

A token, in one sentence

A token is a small piece of text, usually a word or part of a word, that a language model treats as a single unit. Models do not see letters or words the way we do. Before a model reads your prompt, the text is chopped into tokens, and each token is turned into a number the model can work with.

The split is not one token per word. Short, common words are a single token, while longer or rarer words break into several pieces. Spaces and punctuation count too. The word "tokenization", for example, becomes two tokens: "token" and "ization". The phrase "hello world" is two tokens. A rough rule of thumb for English is that one token is about four characters, or roughly three quarters of a word.

Why models use tokens instead of words

Splitting text into whole words has a problem: languages have millions of words, plus names, typos, code, and new coinages a model has never seen. A fixed word list can never cover them all, and anything outside the list becomes an unknown blank.

Tokens solve this by working with pieces. A model needs a vocabulary of only about 50,000 to 200,000 tokens to represent essentially any text, because a word it has never seen can still be built from smaller parts it knows. "Unbelievably" might split into "un", "belie", and "vably". This keeps the vocabulary small while still letting the model handle any input, in any language, including code and symbols.

How tokenization works: byte pair encoding

Most modern models, including the GPT family, tokenize with a method called byte pair encoding, or BPE. The idea is simple. Start with individual characters, then repeatedly find the most common adjacent pair and merge it into a new single unit. Do this thousands of times and the frequent letter groups, common words, and word endings each become their own token, while rare sequences stay broken into smaller parts.

The result is a vocabulary tuned to real language: the words you use constantly are one token each, and the unusual ones cost a few tokens. That balance is why a page of ordinary English lands close to the four-characters-per-token estimate.

A short history

Byte pair encoding did not start in artificial intelligence. It was first described by Philip Gage in 1994 as a simple data compression technique, a way to shrink files by replacing common byte pairs with unused ones.

It found its second life in language two decades later. In 2016, researchers Rico Sennrich, Barry Haddow, and Alexandra Birch adapted BPE for machine translation, using it to split rare words into subword units so translation systems could handle words outside their vocabulary. The approach spread quickly. OpenAI's GPT-2, GPT-3, and later models all tokenize with BPE, and OpenAI released its fast implementation, tiktoken, as the tokenizer behind its models.

Why tokens matter in practice

Once you work with AI models, tokens stop being an abstraction and start affecting real decisions:

• Cost. API pricing is per token, for both the text you send and the text you get back. Fewer tokens means a smaller bill.
• Context limits. Every model has a maximum number of tokens it can consider at once, its context window. Long documents or chat histories can exceed it.
• Speed. Models generate one token at a time, so a longer response takes longer to produce.

Because a word count only roughly predicts a token count, guessing can put you over a limit or under-budget a request. Counting the real tokens removes the guesswork.

See it for yourself

The fastest way to understand tokens is to watch text get split. Paste a sentence into the free token counter, switch between models, and look at the breakdown: you will see exactly where each token begins and ends, and why the number rarely matches the word count.