Tokens and embeddings

What a model really sees, why token counts matter for cost and latency, and how text becomes vectors.

8 min read

An LLM never sees characters or words. It sees tokens: integer IDs drawn from a fixed vocabulary, usually 30k–200k entries. Interviewers ask about tokens because almost every practical constraint (cost, latency, context limits, odd failures) traces back to them.

Tokenization

Modern models use subword tokenizers such as Byte-Pair Encoding (BPE), WordPiece or SentencePiece. BPE starts from bytes or characters and repeatedly merges the most frequent adjacent pair into a new symbol until the vocabulary hits a target size.

The result:

Common words become one token (the, function).
Rare words split into pieces (tokenization → token + ization).
Any string can be encoded, because the fallback is individual bytes. There are no true "unknown words".

A useful rule of thumb for English is roughly 4 characters or ¾ of a word per token. Code, non-English text and numbers usually cost more tokens per character.

Why it matters in practice

Cost and latency scale with token count, and output tokens are generated one at a time, so they dominate latency.
Context limits are measured in tokens, not characters.
Arithmetic and spelling are hard partly because numbers and words are chunked unpredictably. 12345 may be split as 123 + 45.
Languages are not priced equally. A sentence in Hindi or Thai can take several times as many tokens as its English equivalent.

Embeddings

Each token ID indexes a row of the embedding matrix (shape vocab_size × d_model). That row is a learned dense vector, typically 1k–16k dimensions in large models. Training pushes tokens used in similar contexts toward similar vectors.

Because attention has no built-in notion of order, the model also needs positional information. Common approaches:

Approach	Idea	Notes
Absolute (learned or sinusoidal)	Add a position vector to each token embedding	Original Transformer; extrapolates poorly
RoPE (rotary)	Rotate query/key vectors by an angle that depends on position	Used by most modern open models; relative by construction
ALiBi	Add a distance-based penalty to attention scores	Simple, extrapolates to longer inputs

Embedding models vs. LLM embeddings

"Embeddings" in a RAG interview usually means sentence or document embeddings from a dedicated embedding model, trained with a contrastive objective so similar meanings land close together. These are different from the per-token input embeddings inside an LLM, and interviewers like to check that you know the difference.

Interview angle

If asked "Why does the model fail to count the r's in strawberry?", a strong answer mentions that the model sees a few subword tokens rather than letters, so character-level questions require it to recall spelling indirectly.