#From "Speaking Like a Human" to GPT-2: A First-Principles Evolution of Language Models

Series Introduction: Starting from first principles and low-level tensor operators, this series documents the complete journey of building modern Large Language Models (LLMs) from scratch—all the way to Sparse Mixture-of-Experts (Sparse MoE). Part 1 draws inspiration from Andrej Karpathy's Neural Networks: Zero to Hero course, walking through the underlying logic that led from the simplest statistical lookup table (Bigram) to the 2019 GPT-2 architecture.

#1. The Starting Point: How Would You Design an Algorithm That "Speaks Like a Human"?

Setting aside all deep learning jargon, suppose you had to design an algorithm from scratch to automatically generate human language. What would be the most intuitive first iteration?

Collect a massive corpus of text, count which character (or word) is most likely to follow each character, and sample according to those probabilities to piece together a complete response.

For example, after scanning a Chinese corpus, suppose we find that after the character "我" ("I"), the probability distribution of the next character being "喜" (like-), "是" (am), "和" (and), or "椅" (chair) is:

• $P(\text{喜} \mid \text{我}) = 0.001$
• $P(\text{是} \mid \text{我}) = 0.0008$
• $P(\text{和} \mid \text{我}) = 0.00076$
• $P(\text{椅} \mid \text{我}) = 3 \times 10^{-9}$

When sampling from this distribution, the model is far more likely to output "我喜" than "我椅". This is the Bigram model: every time it predicts the next token, it depends only on the 1 current token. If the vocabulary size is $\vert{}V\vert{}$, the parameters of the entire Bigram model are essentially a 2D transition probability lookup table of shape $\vert{}V\vert{} \times \vert{}V\vert{}$.

A Bigram model has the most basic capability of stitching together local phrases. If the probability of "欢" following "喜" is also high, in some sampling run the model can piece together a sensible phrase like "我喜欢" ("I like"). However, because its horizon is only 1 token (a first-order Markov assumption), it is overwhelmingly likely to generate sequences that make sense pairwise locally, yet are completely incoherent as a whole—for example, "大家人们" ("大家" [everyone] is valid, "家人" [family] is valid, and "人们" [people] is valid, yet chained together they make no logical sense).

#2. Expanding the Horizon and the Three Brick Walls of N-grams

To better control the output and eliminate "大家人们"-style gibberish, the most natural second iteration is to include more context when making predictions:

Instead of looking at only the previous 1 character, we use the previous 2 (or more) characters as a prefix. For instance, when the input prefix is "我喜", the conditional probability of the 3rd character being "欢" ($P(\text{欢} \mid \text{我}, \text{喜})$, forming "我喜欢" / "I like") rises sharply, whereas the probability of "庆" ($P(\text{庆} \mid \text{我}, \text{喜})$) plummets—even though "喜庆" ("festive") is a high-frequency word on its own in a Bigram model.

This is the N-gram model. The prefix length referenced each time we predict the next token ($n-1$ tokens) is the model's Context Size. Mathematically, it approximates the chain rule of joint probability over a sequence:

$$P(x_1, x_2, \dots, x_T) = \prod_{t=1}^{T} P(x_t \mid x_{t-(n-1)}, \dots, x_{t-1})$$

Within a small range, increasing $n$ noticeably improves output quality. However, this approach—based on statistical lookup tables over discrete symbols—quickly hits three brick walls and fails to scale up:

1. The Curse of Dimensionality (Exponential Explosion of the State Table): A lookup table must store the combinatorics of every possible prefix. While there are only 26 English letters, a vocabulary $\vert{}V\vert{}$ of Chinese characters or subword tokens easily reaches tens of thousands. Suppose $\vert{}V\vert{} = 10,000$:
   • Bigram ($n=2$) requires storing $10^4 \times 10^4 = 10^8$ floating-point numbers;
   • 4-gram ($n=4$) requires $10^{16}$ entries (~40 PB);
   • Expanding the context window to $T = 4096$, common in modern LLMs, would require $10000^{4096}$ table states.
2. Semantic Islands (Zero Generalization): In a discrete lookup system, each token is merely an isolated integer ID. Even if "I like eating apples" appears ten thousand times in the corpus, as long as the exact combination "I love eating pears" has never appeared, the unsmoothed probability predicted for the latter is strictly 0. Lookup tables cannot share semantic similarity between "like/love" or "apples/pears".
3. Extremely Low Signal-to-Noise Ratio (Dilution by Irrelevant Context): When the context stretches to hundreds of tokens, not every preceding word provides information gain for predicting the next token. An N-gram welds all preceding $n-1$ tokens into a rigid, monolithic lookup key—inserting a single inconsequential filler word in the middle causes the entire prefix match to fail.

#3. Paradigm Shift: From Discrete Lookup Tables to the Transformer

How can we expand the context window (Context Size) to thousands or tens of thousands of tokens without an exponential explosion in parameters, while simultaneously filtering long contexts to extract only the information truly useful for the current prediction?

From N-grams to GPT-2 (Transformer Decoder), the architecture accomplishes four core leaps:

#Step 1: Eliminate Lookup Table Explosion with Continuous Word Vectors (Embeddings)

Instead of indexing a multi-dimensional probability table directly with discrete token IDs, we assign each token in the vocabulary a learnable vector of fixed dimension (e.g., $d = 768$), known as a Token Embedding.

In this continuous vector space, semantically similar tokens ("apple" and "pear") lie close to each other in angle. Predicting the next token shifts from "looking up an exact prefix in an exponentially large table" to "feeding fixed-dimension vectors into a neural network for matrix multiplication." The number of model parameters no longer grows exponentially with context length $T$.

#Step 2: Filter Useful Information via QKV Self-Attention

Suppose the current context contains $t$ tokens, corresponding to the vector sequence $x_1, x_2, \dots, x_t$. When predicting token $t+1$, how should we aggregate information from the first $t$ tokens?

• The Naive Approach (Uniform Averaging / Bag-of-Words): Simply sum the first $t$ vectors and divide by $t$: $y_t = \frac{1}{t}\sum_{i=1}^t x_i$. The result is that critical clues get diluted by filler words into a muddy average.
• The Transformer Approach (QKV Dynamic Weighted Routing): Let the model learn to "pick out what matters." To do this, the Transformer uses three independent weight matrices $W_q, W_k, W_v \in \mathbb{R}^{d \times d_k}$ to project each token into three vector roles:
  1. Query ($q_t = x_t W_q$) — What I am looking for: Represents the features that current position $t$ is querying from the context in order to predict the next token (e.g., if the current token is a transitive verb, it looks back for a subject and modifying adverbs).
  2. Key ($k_i = x_i W_k$) — What I have: Represents the feature label advertised by historical position $i$ ($i \le t$) (e.g., "I am a noun," "I am a temporal adverbial").
  3. Value ($v_i = x_i W_v$) — The actual content I pass along: Projects the original vector $x_i$ linearly into $v_i$, decoupling "address matching" from "content transmission" and expanding the representational capacity.

When predicting the output at position $t$, we take the current query $q_t$ and compute its dot product with the keys $k_i$ of all preceding positions ($1 \dots t$). The larger the dot product, the more important historical position $i$ is for the current prediction. Dividing by the scaling factor $\sqrt{d_k}$ (to prevent large dot products from pushing Softmax into regions with vanishing gradients) and normalizing yields the weight for each historical token, which we use to compute a weighted sum of $v_i$:

$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T + M}{\sqrt{d_k}}\right) V$$

(Note: $M$ is the upper-triangular Causal Mask filled with $-\infty$, ensuring that token $t$ can only attend to history $1 \dots t$ and never peek at future answers.)

#Step 3: Inject Position Information (Positional Encoding)

Pure vector dot products $q_t \cdot k_i$ have a blind spot: they only care about content similarity and completely ignore distance and word order (permutation invariance). Without intervention, the sets of context vectors in "I hit you" and "You hit me" look identical to Self-Attention.

In natural language, however, each preceding token has vastly different importance depending on its relative distance and syntactic position. Therefore, GPT-2 introduces a learnable position embedding table ($W_{pe} \in \mathbb{R}^{T_{\max} \times d}$) at the input layer, adding the token embedding and position embedding directly before entering Attention:

$$x_t = \text{TokenEmb}(\text{idx}_t) + \text{PosEmb}(t)$$

#Step 4: Alternating Communication and Computation (Attention + MLP)

Self-Attention solves cross-position context gathering (Communication), but the gathered result is only a linear weighted combination of historical vectors $V$. To perform deep non-linear processing on the gathered context features, GPT-2 follows each Attention layer with a feed-forward network (MLP / FFN, responsible for per-token feature transformation, i.e., Computation), paired with Residual Connections and LayerNorm, stacking $L$ layers to output the final next-token probability distribution.

#4. Code Mapping: Comparing the Core Tensor Flows Across Three Generations

Looking back across this evolutionary chain, we upgraded the feature extractor three times (full runnable source code in the Colab notebooks below):

1. Bigram Model (Colab): Multiplies a one-hot vector by a $\vert{}V\vert{} \times \vert{}V\vert{}$ weight matrix (equivalent to direct index lookup), with context horizon $T=1$.
2. MLP Model / Bengio 2003 (Colab): Introduces an Embedding table C to eliminate vocabulary dimension explosion, but handles context via hard concatenation view(-1, T * d). While increasing block_size from 3 to 8 improves short-term performance, the first-layer weight matrix W1 grows linearly with window size $T$, and feature weights cannot be shared across positions—still unable to scale to long texts.
3. GPT-2 / Simple LLM (Colab): Combines TokenEmb + PosEmb with MultiHeadAttention + FeedForward to dynamically aggregate context of any length $T \le T_{\max}$.

#5. Summary: Why Is 2019's GPT-2 Still Not Good Enough?

From Bigram to GPT2, combining Embeddings + Positional Encoding + QKV Self-Attention + MLP fundamentally solved the curse of dimensionality and context filtering bottlenecks of N-grams. This is also where Karpathy's build-nanogpt tutorial concludes.

However, viewed from the perspective of today's production LLMs (such as Llama 3, Qwen, and DeepSeek-V3), this 2019 GPT-2 implementation still harbors four serious mathematical and hardware-efficiency bottlenecks:

• Next Post: LLM Tutorial 2 - Stripping Away the "Semantic Understanding" Filter: Computing Piano Music Directly with GPT-2