#Tokenizers and Positional Encoding

Series Introduction: In Part 1, we went from Bigram to GPT-2. Part 2 used the same kind of model to generate piano music, and Part 3 put text and music into one model. Now we return to a simple text task and change the model one component at a time. For the English experiments, we use short stories so we can read the outputs directly.

#1. The Baseline Model

The baseline is a GPT-2-style model with 6 layers, 6 heads, hidden dimension 384, and block_size = 512. We keep these dimensions throughout this post.

For data, we take 1,968 stories, totaling 1,600,163 characters, from TinyStories V2, a dataset of synthetic short English stories. The loader reads complete stories from a fixed dataset revision, removes exact duplicates, and stops after reaching about 1.6 million characters. Every tenth story goes into validation; the rest go into training. The resulting splits contain 1,440,556 training characters and 159,607 validation characters. All three notebooks use the same split.

The task is ordinary next-token prediction on the concatenated text. We start with a character tokenizer: the corpus contains 74 distinct characters, each mapped to an integer. This gives us a 10.85M-parameter baseline.

• Baseline Source Code: llm_tut4_0_nano_gpt_base_en.ipynb (Colab)

After 2,000 steps on a T4 GPU, the recorded training time is 252.9 seconds. Training loss reaches 0.738, and validation loss reaches 0.814. Validation loss is still falling at the end of this run.

We use the same four prompts in each experiment:

Lucy found a small box under her bed. When she opened it,
Tom wanted to fly his kite, but there was no wind.
The little dog was afraid of the water. One day,
"Why is the moon following us?" asked Lily.

Sampling uses seed 42, temperature 0.7, top_k = 30, and a budget of 180 new tokens. For the character model, that means 180 new characters; BPE can produce a longer passage with the same token budget. A sample ending mid-word or mid-sentence may simply have reached that limit.

Here is the character model's first output:

Lucy found a small box under her bed. When she opened it, she found a big, surprise in the room. The box was so happy to see a secret special came out. Finally, the box felt happy to see the box. They played with the box all day long. Th

The model has picked up familiar story phrases, but “a secret special came out” is broken English, and Lucy soon disappears while the box becomes the main character. The kite prompt also drifts immediately:

Tom wanted to fly his kite, but there was no wind. It was a big, red ball scary to blue.

These are recognizable bits of children's stories, but they do not yet form a consistent story. Character tokenization also limits the 512-token context window to just 512 characters, including spaces and punctuation.

#2. Training Our Own BPE Tokenizer

We now start from those same 74 characters and repeatedly merge the most frequent adjacent pair. This is character-based BPE: a token can become part of a word or an entire word. It does not have to correspond to a meaningful piece of English.

The first learned merges include:

he, an, the, ed, to, and, in, re, it, wa, ou, ha, The

Some are whole words, such as the and and; others are fragments, such as re and wa. The regular expression splits letter runs, digits, punctuation, and whitespace before merging, so a word cannot merge with the space or punctuation beside it.

After 4,022 merges, the vocabulary reaches 4,096 tokens. The notebook prints this example:

Demo (57 chars -> 25 tokens):
Lucy /   / found /   / a /   / small /   / box /   / under /   / her /   / bed / . /   / When /   / she /   / opened /   / it / ,

The blank entries between slashes are space tokens. In this sentence, the learned tokens happen to cover every whole word.

Our 1,440,556 training characters now take 638,983 tokens, or about 2.25 characters per token. On average, a 512-token window therefore covers roughly 1,154 characters. The validation split takes 70,692 tokens.

The larger vocabulary also increases the model size from 10.85M to 12.39M parameters. The Transformer blocks are unchanged, but the shared token embedding/output matrix grows by $(4096 - 74) \times 384 = 1,544,448$ parameters.

The notebook learns BPE before splitting the corpus, so validation text contributes to the merge counts. A stricter experiment would learn the merges only from training text. This is a limitation of the current setup.

• BPE Source Code: llm_tut4_1_nano_gpt_bpe_en.ipynb (Colab)

With the same training settings, the BPE run gives:

step    1 (  1.9s) | batch_loss=8.341 | train=6.298 | val=6.290
step  200 ( 26.4s) | batch_loss=2.945 | train=2.888 | val=2.922
step  400 ( 51.9s) | batch_loss=2.433 | train=2.327 | val=2.409
step  600 ( 78.2s) | batch_loss=2.019 | train=1.915 | val=2.039
step  800 (103.9s) | batch_loss=1.745 | train=1.700 | val=1.853
step 1000 (129.5s) | batch_loss=1.602 | train=1.534 | val=1.767
step 1200 (155.3s) | batch_loss=1.438 | train=1.396 | val=1.694
step 1400 (181.2s) | batch_loss=1.437 | train=1.263 | val=1.639
step 1600 (207.0s) | batch_loss=1.274 | train=1.148 | val=1.609
step 1800 (232.7s) | batch_loss=1.192 | train=1.023 | val=1.596
step 2000 (258.5s) | batch_loss=1.047 | train=0.902 | val=1.595

The first part of the box story now reads:

Lucy found a small box under her bed. When she opened it, it was not a big, round thing inside! The box opened the box and found a big, red ball! The ball was so happy and said, "Thank you, friends!" Lucy and the little dog played with the ball all day long.

The phrases are easier to read, but “The box opened the box” still makes little sense, and the little dog arrives without introduction. BPE has not solved the story's consistency.

The dog prompt also gets repetitive:

The dog ran away. The big dog ran after the tree. He ran after the tree. The big dog was surprised. The dog ran up, and the dog was not. The dog ran fast to catch them. The dog was not fast. The dog was happy and happy. The dog was very happy.

#Why Did Validation Loss Go from 0.814 to 1.595?

The two losses are measured per token, and the tokens are different. The character model predicts one character at a time. The BPE model may predict a whole word in one step.

For a rough comparison, we can use the ratio of tokens to characters in the validation split:

$$1.595 \times \frac{70,692}{159,607} \approx 0.706 \text{ nats/char}$$

This is lower than the character baseline's 0.814. It is an approximation: the logged loss comes from five sampled validation batches, while this ratio comes from the entire validation split. An exact conversion for those batches would divide their total negative log likelihood by the number of characters represented by their target tokens. The character and BPE models also see different raw-text windows.

So the higher raw BPE loss does not by itself mean worse predictions. We also cannot attribute the difference entirely to better word representations: BPE gives the model a wider span of text per window and a larger embedding matrix.

#The Same Number of Steps Sees More Text

With 2,000 steps, batch size 32, and context length 512, each run processes:

$$2000 \times 32 \times 512 = 32,768,000 \text{ target tokens}$$

That is about 22.7 training-corpus equivalents for the character model, but 51.3 for BPE. These are randomly sampled windows, not complete sequential passes, but the difference in exposure is still substantial.

The BPE validation loss has almost flattened by step 2,000, while training loss continues to fall. The gap between them is growing, but validation loss has not yet turned upward in this run.

Next, we keep the BPE tokenizer fixed and change the positional encoding.

#3. Positional Encoding: From Absolute (wpe) to Rotary Position Embedding (RoPE)

In original GPT-2, to make Self-Attention aware of word order, the model maintains a learnable absolute positional embedding table wpe of shape (block_size, n_embd) at the bottom input layer, adding token embeddings and position embeddings before entering the Transformer blocks:

$$x_t = \text{wte}(\text{idx}_t) + \text{wpe}(t)$$

This assigns a distinct vector to each absolute index. If a phrase moves from position 10 to position 300, even though the relative distances between words inside the phrase haven't changed at all, the added positional vectors are completely different. A model can learn to handle a phrase at different positions, but wpe has no built-in inductive bias that relative distance is invariant to absolute shifts.

We now replace wpe with RoPE (Rotary Position Embedding), proposed by Su et al. in the RoFormer paper, injecting relative position information directly inside every Attention layer.

Concretely, we remove wpe from the input layer and, prior to computing Self-Attention in each layer, rotate the Query ($Q$) and Key ($K$) vectors of each token (while leaving Value $V$ unchanged).

In our configuration, each attention head has dimension $d = 384 / 6 = 64$. Because 2D rotation operates on coordinate pairs $(x, y)$, we group the 64 features of a head into 32 consecutive pairs: pair $0$ is $(q_0, q_1)$, pair $1$ is $(q_2, q_3)$, ..., up to pair $31$ $(q_{62}, q_{63})$.

If a token sits at position $m$ within the input window ($m = 0, 1, \dots, 511$), we rotate its $i$-th feature pair ($i = 0, 1, \dots, 31$) by angle $m\theta_i$:

$$\begin{pmatrix} \hat{q_{2i}} \\ \hat{q_{2i+1}} \end{pmatrix} = \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix} \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix}$$

That is:

$$\hat{q_{2i}} = q_{2i} \cos m\theta_i - q_{2i+1} \sin m\theta_i$$

$$\hat{q_{2i+1}} = q_{2i} \sin m\theta_i + q_{2i+1} \cos m\theta_i$$

Likewise, for another token at position $n$ in the input window, its Key vector $K$ has each $i$-th pair $(k_{2i}, k_{2i+1})$ rotated by angle $n\theta_i$ using the exact same rule.

Note that $m$ and $i$ index two orthogonal axes:

• $m$ (or $n$) is the token's sequence position within the context window (along the sequence axis, $0 \sim 511$)—this is what encodes word order;
• $i$ is the feature-pair index inside a single attention head (along the head-dimension axis, $0 \sim 31$ out of 64 features). Each pair has its own fixed base angular frequency $\theta_i = 10000^{-2i/d}$: smaller $i$ rotates rapidly with each token step like the second hand of a clock, while larger $i$ rotates slowly like the hour hand, capturing multi-scale positional relationships across different frequency bands.

Denoting the block-diagonal rotation matrix at position $m$ as $R_m$, the dot product between the rotated Query at position $m$ and the rotated Key at position $n$ depends only on their relative angle difference:

$$(R_m q)^T(R_n k) = q^T R_{n-m} k$$

Thus, position enters the attention dot product purely through relative distance $n-m$. Holding the pre-rotation $q$ and $k$ vectors constant, shifting a pair of tokens from positions $(8, 5)$ to $(408, 405)$ leaves their inner product completely unchanged. This assumes $q$ and $k$ stay unchanged. In deeper layers, the surrounding context can change those vectors, so equal relative distances do not guarantee equal attention scores.

Deleting the wpe table also saves $512 \times 384 = 196,608$ parameters, trimming total model size from 12.39M down to 12.19M.

• BPE + RoPE Full Source Code: llm_tut4_2_nano_gpt_bpe_rope_en.ipynb (Colab)

The RoPE run uses the same BPE tokenizer, split, batch size, and model dimensions:

step    1 (  0.0s) | batch_loss=8.142 | train=6.171 | val=6.162
step  200 ( 32.1s) | batch_loss=2.213 | train=2.126 | val=2.182
step  400 ( 65.7s) | batch_loss=1.742 | train=1.681 | val=1.807
step  600 ( 98.6s) | batch_loss=1.519 | train=1.441 | val=1.666
step  800 (131.6s) | batch_loss=1.321 | train=1.243 | val=1.596
step 1000 (164.7s) | batch_loss=1.136 | train=1.044 | val=1.568
step 1200 (197.8s) | batch_loss=0.983 | train=0.873 | val=1.555
step 1400 (230.7s) | batch_loss=0.929 | train=0.703 | val=1.579
step 1600 (263.7s) | batch_loss=0.765 | train=0.568 | val=1.627
step 1800 (296.6s) | batch_loss=0.675 | train=0.451 | val=1.685
step 2000 (329.5s) | batch_loss=0.557 | train=0.344 | val=1.749

Unlike the character-versus-BPE comparison, these two BPE models use the same tokens and sampled validation windows, so their per-token losses can be compared directly.

At step 800, RoPE reaches 1.596, almost the same as the absolute-position model's best recorded loss of 1.595 at step 2,000. RoPE's lowest recorded validation loss is 1.555 at step 1,200, an improvement of 0.040.

After that, the curves separate: training loss falls to 0.344, but validation loss rises to 1.749. By step 2,000, RoPE is worse on validation than the BPE model with absolute positions. This is a clear overfitting pattern.

The full RoPE run takes 329.5 seconds, compared with 258.5 seconds for BPE with absolute positions. It reaches a similar validation loss in fewer steps, but each step takes longer in these runs. These are observations from one run per configuration.

#Step 1,200 and Step 2,000: What Do the Stories Look Like?

The notebook includes saved samples from both training lengths, using the same prompts and sampling settings.

At step 1,200, the box story begins:

Lucy found a small box under her bed. When she opened it, the box opened, it was a fun little bird. Lucy was so happy! She sat down to bed and ate the bird. The bird flew down and landed on the fence. Lucy and the big bird were very happy. They played together all day. The bird was not tired anymore. Lucy and the bird became best friends and played together every day.

Lucy eats the bird, then the bird flies away and becomes her friend. The sentences contain familiar story patterns, but the sequence of events is inconsistent.

At step 2,000, the same prompt gives:

Lucy found a small box under her bed. When she opened it, she found a tiny kitten. The kitten was sad. Lucy was sad. She took the kitten to her room and picked it up in the box. Inside, she found a tiny kitten. The kitten was scared. She ran to the kitten and saw it shouting. The kitten had fallen down into the pot. Lucy was sad, but she knew the kitten was time to do something good.

This version begins more naturally, but Lucy discovers the kitten twice, a pot appears without explanation, and “the kitten was time to do something good” breaks the sentence.

The moon prompt shows another problem. At step 1,200:

"Why is the moon following us?" asked Lily. "I don't know. I don't know. I don't know," Lily said. She took a kiss and put it in a basket.

At step 2,000, the model avoids that particular repetition but soon loses the sentence structure:

Their mom came into the room and saw the mess. She said, "Lily, what did that even though he touched the sky, the moon would come back to him and get hurt the moon. Lily and Ben felt bad for taking the bird from the sky. They went back to their homes and slept on the sea.

Neither version gives a sensible answer to Lily's question. The lower validation loss at step 1,200 means better average next-token predictions on held-out text; it does not guarantee a better story for every prompt.

The notebook preserves outputs from separate executions. Its inference cells use the model currently in memory rather than loading saved checkpoint files. To reproduce the step-1,200 samples, train to 1,200 steps and run inference before training or rerunning to 2,000.

#4. Summary

All three models use the same TinyStories subset and Transformer dimensions. The numbers below come from the saved English runs; the per-character BPE losses use the approximate validation-corpus ratio described above.

BPE lets the same token window cover more text, and several generated phrases become easier to read. Among the BPE runs, RoPE reaches a similar validation loss sooner and gives the lowest recorded loss at step 1,200, while removing 196,608 position-embedding parameters. Continuing to step 2,000 makes its validation loss worse. Across all the samples, repetition, broken sentences, and inconsistent events remain.

Next, we will replace LayerNorm with RMSNorm and the GELU MLP with SwiGLU, changing one component at a time. After that, we will add a KV cache to avoid recomputing the same keys and values during generation.