How Machines Represent Meaning

The Apple Mystery

We already know tokenization turns words into token IDs. But that alone does not explain how a model tells "apple" the fruit apart from "Apple" the company in the same sentence - "I ate an apple at Apple Store." Solving that mystery is what this chapter is about.

Token IDs Don't Contain Meaning

If "Dog" is token ID 8123 and "Grapes" is 8521, are those two numbers being close to each other meaningful? No. Token IDs are just dummy numbers assigned to tokens - no two IDs are related to each other just because they are numerically close. A token ID tells the model which token is present, nothing about what it means.

So if meaning does not live in the token ID, where does it live? That is exactly what vectorization and embeddings solve.

Vectorization

Vectorization is the process of turning information - words, sentences, documents, even images - into long lists of numbers so a computer can work with their meaning and relationships mathematically.

A simple example: rank fruits on sweetness, size, and crunchiness. Apple might become [.7, .4, .8], Banana [.8, .6, .2], Carrot [.2, .5, .9] - one number per dimension. Real embedding models learn thousands of such dimensions directly from the data they were trained on, which is exactly how "Apple Store" and "apple fruit" end up distinguishable: each sense of the word was discussed differently, often enough, in different contexts, for those dimensions to separate them.

Embeddings

An embedding is a learned numerical representation of something that captures its useful relationships with other things - a large array of numbers produced by an embedding model. Every major AI company trains its own embedding models.

King, Queen, and Banana might embed as [.81, .32, .52], [.79, .36, .48], and [.24, .91, .11] - King and Queen sit close together across those numbers, Banana does not. That is the whole idea: associated words end up with embedding values that sit near each other. These values are not handed in by a human - they are learned automatically from data during training, continuously, on huge amounts of GPU compute.

Embedding as Coordinates

Plotting an embedding on just 2 dimensions makes the idea visible: fruits like Apple and Banana cluster in one area, programming languages like JavaScript and Python cluster elsewhere, human roles like Man and Woman sit together, and royalty like King and Queen sit together in yet another area.

Two caveats worth remembering: more dimensions do not mean more intelligence - higher dimensions capture more complex patterns but cost more storage and computation - and these numbers keep getting readjusted with every round of training.

Semantic Similarity

"How do I center a div?" and "How can I align an HTML element in the middle of its parent?" use almost none of the same words, yet mean the same thing. Semantic similarity is exactly this: a way for computers to check whether two pieces of text mean the same thing, even when the words are completely different.

This was a genuinely hard problem before embeddings, precisely because of endless word-combination possibilities. Embeddings make it tractable by letting a system compare broader meaning instead of only matching identical words.

Cosine Similarity

The actual technique used to measure semantic similarity is called cosine similarity: it looks at the angle between two vectors (arrow representations) rather than their raw values. Vectors pointing in a similar direction represent similar concepts; vectors pointing in unrelated directions represent less similar ones.

Similarity Is Not Truth

It is worth being explicit about a limitation here: two false statements can be semantically similar. Two harmful instructions can be semantically similar. Two texts can be close in topic while completely disagreeing with each other.

Embeddings capture relationships. They do not independently verify facts, intent, quality, or safety - remembering that distinction matters once you start relying on similarity search for anything real.

Token Embedding vs Text Embedding

Both a token and a full piece of text can have their own embedding, and it is worth not confusing the two. Token embedding relates a meaningless token ID to a dense numerical vector, becoming the model's initial internal representation of that token - a representation that can still shift later as the model processes surrounding context.

Text embeddings (also called sentence or document embeddings) work at a different scope entirely - condensing an entire sentence, paragraph, or document into a single vector, mainly used for tasks like search, retrieval, clustering, recommendation, classification, and duplicate detection. Think of it like a book: a token embedding is a definition for one word, while a text embedding is the summary printed on the back cover.

Positional Embedding

"Dog bites Man" and "Man bites Dog" use exactly the same words - the embedding for "bites" would even start out the same in both - yet the two sentences mean completely different things. Word order clearly matters, and token or word embeddings alone don't capture it.

Positional embedding is the technique that gives a model a sense of order and sequence: it tells the model where a token appears, on top of what token embedding already tells it about which token is present. Different model architectures implement this using their own architecture-specific techniques.

How LLMs Handle Context

Take "Java is a language" and "Java is an island in Indonesia." The word "Java" can start out with the exact same initial token embedding in both sentences. But as a Transformer-based model processes each token together with its surrounding tokens, through multiple layers, that representation becomes contextualized - it changes based on everything around it. This is what "contextualization" means: giving the model enough surrounding clues to understand what you mean, rather than guessing from an isolated word alone.

Polysemy - a single word carrying multiple distinct meanings depending on the sentence - is exactly the phenomenon this solves for. It is also worth knowing that bias can creep in here too: if a model was trained on one-sided data, its contextual representations inherit that bias. Modern LLMs typically combine this with hybrid search - keyword search plus embedding search together - for better real-world accuracy.

Common Misconceptions About Embeddings

An embedding is not a dictionary of tokens - it is a numerical representation capturing learned relationships. No single dimension has one clean human-readable meaning - meaning is distributed across many dimensions at once. Similar embeddings do not mean identical meaning - they can indicate topic similarity, association, opposition, category membership, or shared context.

Embedding similarity does not prove a statement is true - similarity measures relatedness, not truth. One embedding model does not work equally well for every task - performance depends on language, domain, data type, text length, and training objective. A visual cluster does not perfectly represent the original space - 2D and 3D projections lose real information. And a larger embedding dimension is not automatically better - bigger representations trade off storage, latency, cost, and quality.