2  Information Representation

Central question: How does a machine represent meaning?

2.1 Introduction

In the previous module, we situated LLMs as systems that learn from text. However, one question was left open, one that the connectionist tradition presupposes but does not answer on its own: how can a computer, which operates only on numbers, handle words, sentences, and, ultimately, meaning? A neural network does not see letters or understand languages; it multiplies and adds numbers. For text to enter this machinery, it must first be converted into numbers, and this conversion cannot be arbitrary: it must preserve, in some way, the relationships of meaning between words.

This chapter deals precisely with that bridge between language and numbers. We will see how text is sliced into units called tokens, how each token is transformed into a list of numbers called an embedding, and how the set of these numbers defines a vector space in which geometric proximity comes to reflect proximity of meaning. By the end, the central question, which asks how a machine represents meaning, will have a concrete answer: meaning, for the machine, is a position in a space of many dimensions.

This answer has an important limit, which we state upfront. We will show what this representation is and what it captures, but the mechanism by which these numbers are actually learned is left for the next module. The order is deliberate: it is easier to understand what a neural network adjusts after first seeing what needs to be represented.

2.2 Learning Objectives

  • Explain why text must be converted into numbers before entering a neural network.
  • Describe what tokenization is and why subwords are used instead of whole words.
  • Define embedding and understand that each token corresponds to a point in a space of many dimensions.
  • Interpret the distributional hypothesis and how it allows the geometry of the space to capture relations of meaning.

2.3 Guiding Questions

  • Why is it not enough to number the words of a dictionary from 1 to N to represent them?
  • What is gained by breaking a rare word into smaller pieces?
  • How can two words with similar meaning end up close in a purely numerical space?
  • In what sense can a “direction” in this space correspond to a relation like “plural” or “feminine”?

2.4 Conceptual Map of the Chapter

Figure 2.1 shows the path a text takes until it becomes a representation that the neural network can process: from raw text to tokens, from tokens to embeddings, and from these to the vector space where meaning resides.

Figure 2.1 - Conceptual map: from text to the semantic vector space

Text Tokenization (subwords) Embeddings (vectors) Vector space semantic

Source: prepared by the author (2026).

2.5 Tokens: the Building Blocks of Text

2.5.1 The Problem: Meaning Is Not an Obvious Number

Before discussing representations, we must clarify what we will represent: a token. A token is the smallest unit of text that the model processes: for now, think of it simply as a word. The model receives a sequence of tokens and must convert each one into a number in order to operate on them.1

Here the first problem appears. The obvious temptation is to assemble a dictionary and assign each word an index: “company” → 1, “startup” → 2, “university” → 3. This fails for a deep reason: an index is a label disguised as a number. The difference between indices means nothing semantically.

Think of an identity document number. Two people can have consecutive numbers and have absolutely nothing in common: different city, different profession, different education level. The number exists, but says nothing about who that person is. Likewise, assigning “company” the index 1 and “startup” the index 2 does not tell the model that these two words inhabit similar worlds. And the reverse also holds: “company” (1) could end up numerically closer to “university” (3) than to “corporation” (10), not by meaning, but by mere ordering chance.

What we need is the opposite: a representation in which numerical proximity reflects proximity of meaning. Similar words should be close; words from distinct worlds should be far apart. Figure 2.2 illustrates this contrast.

Figure 2.2 - Numerical index vs. context-based representation

business
higher education
company1

Neighbors in the text: employee, product, market, investment, customer.

startup2

Neighbors in the text: market, investment, product, customer, employee.

university3

Neighbors in the text: student, professor, curriculum, research, campus.

master’s4

Neighbors in the text: dissertation, advisor, defense, scholarship, graduate studies.

corporation10

Neighbors in the text: market, employee, product, investment.

Use the button above to reorganize the words by context, and click on each word to see which others it usually appears with in the text.

Source: prepared by the author (2026). Illustrative neighborhoods.

It is clear, then, what we want: a representation in which distance reflects meaning. But an uncomfortable question arises: how does a machine, which only sees text, know that “company”, “startup”, and “corporation” are more similar to one another than to “master’s” or “salary”? No one programmed this similarity. There is no table of meanings we can consult. The answer comes from linguistics.

In 1957, J. R. Firth summarized in one sentence what we now call the distributional hypothesis: “you shall know a word by the company it keeps” (Firth, 1957). The intuition is powerful: the meaning of a word is not inside it, but in the contexts in which it appears. If we observe millions of sentences, we will notice that “company” and “startup” appear surrounded by very similar words (employee, product, market, investment, customer), while “master’s” appears alongside completely distinct words (dissertation, advisor, defense, scholarship, graduate studies). This difference in neighborhoods is a robust statistical signal: two words that frequent the same contexts probably describe similar realities.

The distributional hypothesis has a direct and practical consequence: if we count which words each token appears with in a massive corpus, we can build a numerical representation in which proximity captures that similarity of contexts. Words with similar neighborhoods will receive similar representations; words with distinct neighborhoods will receive distinct representations. This is, in essence, what an embedding does: the topic of the next section.

2.5.2 Tokenization: Slicing Text into Units

Before becoming a number, text must be divided into discrete units, the tokens. The design question is: what is the ideal size of each piece? There are two extreme approaches, and it is worth understanding why neither is used in isolation.2

The first approach would be to use whole words as the unit. This creates two problems. First, the vocabulary of a language is practically unlimited, including proper names, slang, technical terms, and compound words, so that no fixed list would suffice. Second, and more serious: a word that never appeared in training simply has no corresponding token, and the model has nothing to do with it.

The solution adopted by most LLMs takes a middle path: subword tokenization, popularized by the Byte-Pair Encoding (BPE) technique in the context of machine translation (Sennrich et al., 2016). The logic resembles assembling a word from building block pieces: large, ready-made pieces for the most common blocks of the language, small and combinable pieces for everything rare or unknown. Common words become a single token; rare, long, or unseen words are broken into already known subwords, so that any text, even with a never-before-seen word, can still be written with pieces from the vocabulary. Figure 2.3 illustrates the process.

Figure 2.3 - Tokenization: text is sliced into tokens, with rare words broken into subwords

Input text inteligência artificial Tokenizer finite vocabulary intel ig ência artificial 4521 88 2090 7720 rare word: 1 word becomes 3 subwords common word: 1 token Each token receives a numeric identifier (an index in the vocabulary). Thus, any text, even with unseen words, can be written with known pieces.

Source: prepared by the author (2026), based on Sennrich et al. (2016). Tokens and identifiers are illustrative.

2.5.3 What Tokenization Causes Day to Day

Tokenization seems like a technical detail, but it explains several behaviors that confuse LLM users. It’s worth knowing them, because they are a common source of frustration and wrong conclusions about what the model “knows.”

The most famous case is letter counting. Ask an LLM to count how many times the letter “r” appears in “strawberry” and it commonly gets it wrong. The reason isn’t stupidity: the model doesn’t see individual letters. It sees the word sliced into a few subword tokens (something like “st”, “raw”, “berry”), and never had access to the characters one by one. Asking it to count letters is asking it to describe something its representation erased, like asking someone who counts syllables to say how many atoms are in a word.

The second consequence is economic, and is revisited in Module 13: different languages spend different numbers of tokens for the same content. Tokenizers were optimized mostly for English, so Portuguese and Spanish (and even more so languages with other alphabets) tend to be sliced into more tokens to say the same thing. Since you pay per token, the same text can cost more in Portuguese or Spanish than in English, and take up more space in the context window.

The third shows up with numbers and dates. A large number like “1,048,576” can be broken into several tokens in unintuitive ways, which helps explain why models sometimes get arithmetic wrong with long numbers: the number’s own representation is fragmented in a way that doesn’t correspond to its value. It’s one more reminder that exact calculation is a job for a tool (Module 10), not for the model’s guesswork.

Figure 2.4 makes the second consequence, the economic one, concrete: the same question, written in Portuguese/Spanish and in English, spends different numbers of tokens.

Figure 2.4 - The same sentence costs more tokens in Portuguese than in English

Same question, same meaning, different token counts 11 tokens Portuguese “Qual é a capital da Austrália?” 7 tokens English “What is the capital of Australia?” Illustrative counts: they vary by tokenizer and model, but the pattern (non-English > EN) repeats.

Source: prepared by the author (2026). Illustrative counts from a commonly used tokenizer.

The numeric identifier of each token, however, still suffers from the same flaw as the naive dictionary: it is an arbitrary index, unrelated to meaning. It serves only to locate the token. The next step is what actually carries meaning.

2.6 Embeddings: Meaning in Vectors

2.6.1 From Token to Vector: the Embedding

To each token, the model associates an embedding: a list of real numbers, that is, a vector, with tens to thousands of positions. Think of a registration form that describes a person by numbers: height, age, years of education, income. Each characteristic occupies a position on the form, and two people with similar forms tend to resemble each other in several respects. An embedding does something analogous with words, only with hundreds or thousands of “characteristics” instead of four, and these characteristics are not defined by us: they emerge from training.3

The idea was consolidated in neural language models (Bengio et al., 2003) and became central with techniques such as word2vec (Mikolov, Chen, et al., 2013) and GloVe (Pennington et al., 2014). This vector is not assigned by hand; each of its positions is a number adjusted during training, and the set defines the token’s location in a space of many dimensions. Figure 2.5 shows the conversion.

Figure 2.5 - Embedding: each token becomes a vector of numbers, its position in a space of many dimensions

Each token receives a vector (embedding): a list of real numbers cat 0.12 -0.84 0.33 0.07 -0.21 … king 0.77 0.05 -0.40 0.61 0.18 … d dimensions (e.g. from 300 to several thousand) The list of numbers is the token’s position in a space of many dimensions. The values are not chosen by hand: they are learned (see Module 3).

Source: prepared by the author (2026). Illustrative values.

2.6.2 The Semantic Vector Space

When all tokens become vectors, they come to inhabit the same vector space: a geometric space of many dimensions in which each token is a point. What makes this space remarkable is that, trained according to the distributional hypothesis, it organizes the points so that geometric proximity reflects proximity of meaning.4 Words with similar meaning form neighborhoods, and, more surprisingly, certain directions in the space come to correspond to systematic relations.

Figure 2.6 - Semantic vector space: proximity is meaning, and directions encode relations

2D projection of a space with hundreds of dimensions (illustrative only) cat dog horse animals man woman king queen same direction = “feminine” relation vector(king) − vector(man) + vector(woman) ≈ vector(queen)

Source: prepared by the author (2026), based on Mikolov et al. (2013). Illustrative positions.

2.6.3 Static and Contextual Embeddings

There is an important limitation in the classic embeddings we’ve described so far, and overcoming it was one of the leaps that led to modern LLMs. In word2vec and GloVe, each word has a single vector, fixed, the so-called static embedding. This creates a problem with words that have more than one sense. The word “bank” gets the same vector in “I sat on the river bank” and in “I went to the bank to withdraw money”, even though they mean completely different things. The static embedding is forced to blend the two senses into an average, which serves neither well.

The solution is contextual embeddings: instead of a fixed vector per word, each token’s vector is recalculated based on the surrounding words. Under this scheme, “bank” gets a vector pulled toward the world of rivers when surrounded by “sat” and “river”, and another, pulled toward the financial world, when surrounded by “withdraw” and “money”. This is exactly what the Transformer architecture does, through the attention mechanism, and it is the topic of Module 4. For now, keep the distinction in mind: the embeddings in this chapter are the starting point (the initial table that associates a vector with each token), and it is the network that, layer by layer, transforms them into context-sensitive representations.

Important

Real case: the bias that lives in the vectors. The same geometry that produces the lovely analogy “king − man + woman ≈ queen” also produces others, considerably less innocent. Bolukbasi and collaborators showed, in 2016, that embeddings trained on large web texts answered “man is to computer programmer as woman is to…” with “homemaker” (Bolukbasi et al., 2016). The model didn’t invent this stereotype; it inherited it from the human text it was trained on, and crystallized it in the geometry of the space. The lesson is twofold and runs through the book: first, embeddings capture not only meaning but also the biases of the society that produced the texts; second, “learned from data” is never synonymous with “neutral.” This is one of the reasons why curating training data (Module 5) and careful evaluation (Module 8) matter so much.

Transition note (for Module 3): This chapter shows what embeddings are and what they capture, but not how these vectors are obtained. The values of each position are not assigned manually: they emerge from the adjustment of parameters of a neural network exposed to enormous amounts of text. This learning mechanism, what “adjusting parameters” means and how it makes the vectors organize themselves, is precisely the topic of the next module.

2.7 In Practice: Observe Tokenization and the Vector Space

Two free experiments make the chapter’s two central concepts visible.

Experiment 1: slice text into tokens. Visit an online tokenizer (Tiktokenizer, at tiktokenizer.vercel.app, shows the colored slicing instantly). Paste a sentence and watch how it’s broken into tokens; notice that common words become a single token and rare or compound words split into pieces. Now do the test that reveals Module 13’s economics: write the same sentence in Portuguese/Spanish and in English and compare the count. Non-English almost always spends more tokens. Finally, paste a word like “strawberry” or a long number and see, with your own eyes, why the model struggles to count its letters: it never sees them separately.

2.8 Analogies: the Map Where Nearby Words Have Similar Meanings

A useful analogy compares the vector space to a map. In a city, places with similar functions tend to be close, and directions on the map correspond to real relations: going “north” means the same thing starting from any point. In the embedding space, words with similar meaning are close, and a direction can correspond to a relation such as “feminine” or “plural”, applicable to several pairs of words. The analogy works to convey the intuition of proximity and direction. It breaks down in two respects. First, a city map has two dimensions, whereas the embedding space has hundreds or thousands, and our visual intuition does not keep up with this dimensionality.5 Second, the positions were not drawn by a cartographer with a plan: they emerged statistically from the data, and therefore may contain distortions and biases inherited from the training text, a point revisited in the modules on limitations and data.

2.9 From Word to Vector, in a Diagram

Figure 2.7 summarizes the complete representation pipeline, from raw text to the actual input of the neural network. Each stage is a transformation: tokenization discretizes the text, the embedding table replaces each identifier with a vector, and the result is the sequence of vectors that feeds the subsequent layers.

Figure 2.7 - Representation pipeline: from text to the sequence of vectors

Raw text Tokens (subwords) Identifiers (indices) Embedding table Sequence of vectors Neural network input

Source: prepared by the author (2026).

The point to retain from the diagram is that the representation is a sequence of transformations without loss of identity: each token maintains its position in the sequence, and it is precisely this order that the next step of the architecture will need to take into account.

2.10 Key Concepts

  • Token: smallest unit into which text is sliced for processing; can be a word, a subword, or a character.
  • Tokenization: process of dividing text into tokens; the use of subwords (BPE) ensures a finite vocabulary and coverage of unseen words (Sennrich et al., 2016).
  • Embedding: vector of real numbers associated with a token, which defines its position in a space of many dimensions (Bengio et al., 2003; Mikolov, Chen, et al., 2013).
  • Semantic vector space: geometric space in which proximity between vectors reflects proximity of meaning.
  • Static vs. contextual embeddings: a fixed vector per word (word2vec) versus a vector recalculated according to context (what the Transformer does, Module 4).
  • Bias in embeddings: stereotypes inherited from training texts that become fixed in the geometry of the space (Bolukbasi et al., 2016).
  • Distributional hypothesis: principle according to which the meaning of a word emerges from the contexts in which it occurs (Firth, 1957; Harris, 1954).

2.11 Myths: “AI Reads Letters” and “Counts Characters”

The first misconception is confusing the token identifier with its representation of meaning. The index in the vocabulary is merely an arbitrary address; what carries meaning is the embedding, the vector associated with that index.

The second is imagining that each embedding dimension has a nameable meaning, such as “one dimension for gender, another for number”. In general, the axes are not interpretable in isolation; meaning is distributed across the set of dimensions, and only certain combined directions correspond to recognizable relations.

The third is assuming that the analogy “king minus man plus woman” works with perfect and universal precision. This is an approximate regularity, frequently observed, but subject to failures, especially outside the most common pairs and with polysemous words. The regularity illustrates the phenomenon, not an exact law.

2.12 Connection to the Next Chapters

We have seen what the embedding representation is and what it captures. Module 3 opens the missing mechanism: what a neural network is and what it means, mechanically, to “learn” these vectors by adjusting parameters. With representation and learning in hand, Module 4 will show how the Transformer architecture uses these vectors, taking into account the sequence order and the relations between tokens, to produce contextual understanding.

2.13 Going Further

To do (try it and observe)

  • Tiktokenizer (tiktokenizer.vercel.app): slice sentences into tokens, compare the count in your language and in EN, and see why the model gets letter counting wrong.
  • Embedding Projector (projector.tensorflow.org): navigate in 3D through a real embedding space and inspect a word’s neighbors.
  • Jay Alammar, The Illustrated Word2vec: a visual, intuitive explanation of how words become vectors.

To read (deepen the fundamentals)

  • MIKOLOV, T. et al. Efficient Estimation of Word Representations in Vector Space. 2013. The article that popularized word embeddings and their regularities.
  • BOLUKBASI, T. et al. Man is to Computer Programmer as Woman is to Homemaker?. 2016. The study that exposed the gender bias built into embeddings.
  • FIRTH, J. R. A Synopsis of Linguistic Theory. 1957. The origin of the distributional maxim that underpins modern representation.
  • SENNRICH, R.; HADDOW, B.; BIRCH, A. Neural Machine Translation of Rare Words with Subword Units. 2016. The introduction of subwords (BPE) that made vocabularies finite and robust.
  • BENGIO, Y. et al. A Neural Probabilistic Language Model. 2003. The model that consolidated the idea of representing words as learned vectors within a neural network.
  • PENNINGTON, J.; SOCHER, R.; MANNING, C. D. GloVe: Global Vectors for Word Representation. 2014. An alternative to word2vec that builds embeddings directly from global co-occurrence statistics.

  1. The mathematical operations that the neural network applies (matrix multiplications, sums, activation functions) require all input data to be numeric values. Text, as we write it, is a sequence of characters: it cannot directly enter these operations. Tokenization and embedding are the bridge that converts characters into numbers the network can process.↩︎

  2. At the other extreme lies character-level tokenization, in which each letter is a token. The vocabulary becomes tiny (a few dozen symbols) and no word is unknown, but the price is a much longer sequence to represent the same text, which burdens processing. Subword tokenization seeks the balance between these two extremes: manageable vocabulary and sequences of reasonable length.↩︎

  3. Unlike the registration form, in which each position has a clear label (height, age), the positions of an embedding rarely correspond to an isolated, nameable concept. Meaning is distributed across the set of positions, and only combinations of them, not a single isolated position, tend to correspond to recognizable relations, as we will see in the next section.↩︎

  4. This “proximity” can be measured in different ways, with Euclidean distance and cosine similarity (which compares the angle between two vectors, ignoring their magnitude) being the most common. The choice of metric does not change the central idea: similar vectors represent similar meanings. The classic example, observed in word2vec, is the regularity that approximates the vector of “king” minus that of “man” plus that of “woman” to the vector of “queen” (Mikolov, Yih, et al., 2013). Figure 2.6 illustrates the idea in a two-dimensional projection.↩︎

  5. This mismatch between intuition, formed in a three-dimensional world, and the real behavior of high-dimensional spaces is known as the “curse of dimensionality”: as the number of dimensions grows, intuitive notions such as “near” and “far” begin to behave in ways that defy geometric common sense.↩︎