The Secret Language of LLMs

Computers Don't Understand Words - They Understand Numbers

Programming languages like C and C++ exist because computers cannot understand plain English directly. The same is true for LLMs: a sentence like "These notes are awesome" cannot be handed to a model as-is. What an LLM understands is numbers.

The sentence first gets broken into pieces - These, notes, are, awesome - and each piece is converted into a number, producing an array like [78, 12, 23, 433] that is what actually gets passed to the model.

Tokens, Token IDs, and Tokenizers

When a sentence is broken into words or word-pieces, each piece is called a token, and the number assigned to a token is its token ID. The sequence of token IDs is what the LLM actually reads and processes.

This also flips how prediction works: an LLM is not really predicting the next word directly - it is predicting the next token ID, which is then mapped back to a token for you to read. The tool that does this splitting is called a tokenizer, and there are multiple tokenizer types in use across the industry, chosen per model and per task - this is a core part of both training and inference.

One word is not guaranteed to become exactly one token. "Awesome" might stay as one token, or it might split into two, like "awe" + "some" - it depends entirely on the tokenizer's vocabulary.

Subword Tokenization

You can see this live on tools like the OpenAI tokenizer or tiktokenizer. Feed in a word like "untrustable" and it may split into three tokens - "un", "trust", "able" - because those pieces are common and reusable across many other words. This is subword tokenization: breaking uncommon or longer words into smaller, reusable chunks instead of storing every possible word as its own token.

Confusingly, this is not always predictable from meaning or grammar. A word like "undone" might stay as a single token even though "un" + "done" seems like the obvious split - simply because "undone" already exists as one entry in that tokenizer's vocabulary. The same word can be split completely differently by a different tokenizer. Tokenization follows the vocabulary the tokenizer was trained on, not the grammatical structure of a word.

Vocabulary: The Fixed List Behind Every Token ID

A tokenizer's vocabulary is the fixed list of every unique word, subword, or character it recognizes and can convert into a number. Every token ID comes from a lookup against this vocabulary mapping.

Different companies use different tokenizer families entirely: OpenAI models use variants of Byte Pair Encoding (BPE), categorized by vocabulary size, while Meta's Llama models use SentencePiece-BPE. A larger, better-trained vocabulary generally makes tokenization more efficient - which matters a lot once you start thinking about cost and multilingual support.

Byte Pair Encoding (BPE)

BPE is the tokenizer family behind OpenAI's models. The idea: a word like "low" might get its own token ID, and that same token can be reused as part of "lower" and "lowest". The same pattern applies to prefixes like "un" showing up across "unbreakable", "untrustable", "unmanageable", and more.

Since every token can ultimately be represented as bytes (combinations of 0s and 1s), a BPE tokenizer starts from small byte-level units and learns which neighboring pairs occur together often enough to be worth merging into a single new token - growing the vocabulary over time. Other tokenizer families like WordPiece and Unigram take a similar spirit but with different merging rules; the common goal across all of them is a vocabulary that makes tokenization efficient.

English vs Other Languages

Tokenization is purely about text patterns - there is no meaning or emotion baked in, so different languages routinely produce very different token counts for what is functionally the same sentence. Writing something in Hinglish (mixed Hindi-English, like "Mai Artificial Intelligence seekh raha hu") does not tokenize the same way as writing the equivalent pure-English sentence, even though the meaning is identical to a human reader.

In practice, English tends to be the most token-efficient language for most current tokenizers, while languages like Hindi or mixed Hinglish can cost noticeably more tokens for the same idea. Vocabulary size and training coverage are the main levers that determine how efficiently a model handles a given language - newer tokenizer versions have gotten meaningfully better at this than older ones.

Tokenization Fertility

Tokenization fertility is the average number of tokens an AI produces per word. Lower fertility means fewer tokens per word - faster, cheaper, and able to fit more text into a single context window. Higher fertility means a word gets chopped into many tiny fragments, which increases cost and fills up the context window faster - something that shows up disproportionately in some non-English languages.

Emojis, code, special characters, capitalization, and even plain spaces all affect tokenization too - adding or removing a single space, or changing a dash, can change the token count. Every distinct text pattern maps to its own token ID.

System Instructions vs User Instructions

Beyond the words you type, models also work with special tokens tied to system instructions - a kind of hidden rulebook that tells the AI how to behave, separate from what you actually ask. A user instruction, by contrast, is simply the specific question or task you type in.

How exactly system and user prompts get combined and passed to the model differs from one LLM and company to another.

The Context Window

A context window is the amount of tokenized information a model can process within a single request or generation - in plain terms, the short-term working memory of an AI: the maximum text it can "see" and remember at any one moment.

That window is shared between everything you send and everything the model generates in return - and crucially, it includes far more than just your typed message: system instructions, attached documents, prior text, and tool outputs all count against the same limit.

When a context window threatens to overflow, the practical fixes are to truncate input, summarize earlier messages instead of replaying everything, and split unrelated work across separate chats rather than cramming every task into one long conversation. It is also worth remembering that a model does not perfectly remember an old conversation forever - it works from a summary of it, and only while that context stays live.

Common Misconceptions About Tokens

A short list worth keeping in mind: one token is not always one word. Not every company or model uses the same tokenizer. A token ID does not represent meaning on its own. A single visible emoji is not guaranteed to be one token. A larger vocabulary is not automatically better. A larger context window does not guarantee perfect memory. And more tokens in a prompt do not automatically produce a better result - a longer prompt is not the same thing as a better prompt, and every extra token has a real, direct effect on API cost.