acytoo

LLM Tutorial 2 - Stripping Away the "Semantic Understanding" Filter: Computing Piano Music Directly with GPT-2

#Stripping Away the "Semantic Understanding" Filter: Computing Piano Music Directly with GPT-2

Series Introduction: In Part 1 (From Bigram to GPT-2), we derived the architectural evolution from Bigram models to GPT-2. When people encounter terms like Self-Attention and Token Embeddings, it is easy to view them through a sci-fi lens—as if the neural network truly "understands" the semantics of human language. In reality, under the hood it is entirely tensor operations and conditional probability fitting. Today, without changing a single line of model architecture code, we swap the training dataset from Shakespearean text to MIDI piano pieces, train for 10 minutes on a free Colab T4 GPU, and see what the exact same Decoder-Only Transformer computes.

#1. Core Demystification: A Transformer Has No Idea Whether the Input Is Text or Piano Keys

Recall the SimpleLLM we implemented in Part 1 (a 6-layer, 6-head GPT-2 architecture with hidden dimension 384 and context window 256):

  1. Input: Only accepts a 1D integer sequence of shape (B, T).
  2. Output: Only predicts the probability distribution of the next integer ID of shape (B, T, vocab_size).

To a Transformer, whether the integer 39 represents an English letter, the Chinese character "我" ("I"), or Middle C (C4) on a piano makes zero mathematical difference. It understands neither grammar nor music theory. As long as information from any modality can be encoded into a 1D stream of discrete integers (a Token Sequence), the exact same autoregressive matrix multiplications will brute-force fit its long-range co-occurrence probabilities.

#2. How Do We Turn Piano Music into a Token Sequence?

Text is naturally 1D and linear, whereas piano music is polyphonic: left-hand chords and right-hand melodies may be struck simultaneously, and each note varies in duration and key velocity.

To feed music directly into the exact same GPT-2 model, we decompose MIDI music into 308 discrete events (vocab_size = 308), flattening an entire piano performance into a 1D integer stream:

  • 0 .. 87 (Key press NOTE_ON, 88 tokens): Corresponds to the 88 keys on a piano (MIDI pitches 21 to 108).
  • 88 .. 175 (Key release NOTE_OFF, 88 tokens): Releases the corresponding piano key. Emitting multiple NOTE_ON tokens consecutively without advancing time decodes into a chord struck simultaneously.
  • 176 .. 275 (Time advance TIME_SHIFT, 100 tokens): Advances the timeline forward by 1 to 100 steps (10 ms per step, advancing up to 1 second in a single token).
  • 276 .. 307 (Key strike velocity VELOCITY, 32 tokens): Quantizes MIDI velocity (0 to 127) into 32 bins to record how hard a key is struck.

After this encoding transformation, an expressive piano solo looks to the model like a pure integer sequence: [298, 20, 275, 51, 44, 39, ...] (set velocity $\to$ press bass root note $\to$ advance 1 second $\to$ press a triad chord).

#3. Experimental Comparison: 10 Minutes on a Free T4 GPU, Two Contrasting Styles

To visually—and aurally—test how much this "pure probability computation" can learn, we selected two stylistically contrasting piano repertoires, trained two instances of the exact same SimpleLLM architecture (~10.8M parameters) for 1,000 steps each (taking ~8–10 minutes), and at inference time fed the exact same starting token (a single Middle C, token 39), letting each model autoregressively generate 1,000 event tokens:

#0. Control Baseline: Untrained Random Weights

When parameters are randomly initialized, the model has no concept of pairing NOTE_ON with NOTE_OFF or pacing time steps, smashing keys randomly across octaves on all 88 keys—sounding like a cat walking across the keyboard:

#1. Slow Style: Erik Satie's Melancholic Adagios

Trained on 6 slow-tempo pieces by Erik Satie (such as Gymnopédies and Gnossiennes, totaling 8,211 tokens). After 1,000 training steps (loss down to 0.0337), the model learned a high-probability distribution over large TIME_SHIFT tokens, naturally creating breathing room and space, with slow alternations between left-hand bass notes and right-hand triads:

#2. Bright Style: Scott Joplin's Upbeat Ragtime

Trained on 6 Ragtime pieces by Scott Joplin (such as The Entertainer and Maple Leaf Rag, totaling 41,524 tokens). Also after 1,000 training steps (loss down to 0.0537), the generated sequence is densely packed with short 10–20 ms time steps and rapid note transitions, directly reproducing the bouncy syncopated groove of Ragtime:

#4. Summary: There Is No "Musical Understanding"—Only Statistical Regularities

Comparing the three audio clips above leads to a straightforward conclusion:

  1. No Semantics, Only Distributions: From start to finish, not a single line of code in the model encodes rules about sheet music, harmony, or metronomes. What we perceive as "melancholic slow pacing" versus "upbeat syncopation" is, at the lowest level, merely a difference in conditional probability distributions between time tokens (176..275) and pitch tokens (0..175).
  2. The Essence of Attention: When generating the next note, QKV Self-Attention simply performs vector dot-product retrieval over the preceding 256 historical integers—for instance, noticing that a Middle C NOTE_ON (Token 39) appeared earlier without a matching NOTE_OFF (Token 127), and consequently boosting the output logits for TIME_SHIFT and NOTE_OFF (Token 127).
  3. Memorization and Stitching in the Small-Data Regime: Because the training set contains only tens of thousands of tokens while the model has over ten million parameters (driving training loss down to 0.03 ~ 0.05), the model effectively memorizes local motivic fragments from the original pieces and recombines them. Moving from "rote memorization of local patterns" to genuine long-range generalization requires scaling up both data size and model capacity—which brings us directly to the modern LLM architectural evolution we will unpack next.

See Also: