Tokenization

BPE, WordPiece, and how text becomes token IDs

11.1 What You Will Learn

In Chapter 10 we built an embedding layer that takes a category ID and returns a learned vector. But we never addressed where those IDs come from in the first place. If the input is text, you need some way to break a string of characters into pieces and assign each piece a numerical ID before the embedding layer can do anything with it. That process is tokenization.

There are many ways to split text into pieces, and the choice matters more than you might expect. You could treat every character as its own token, which is simple but produces very long sequences. You could split on spaces and treat every word as a token, which is compact but creates an enormous vocabulary and cannot handle words it has never seen before. Or you could do something in between, breaking text into subword chunks that balance sequence length against vocabulary size. In this chapter we build a tokenizer from scratch, starting with single character tokenization and working up to Byte Pair Encoding, which is the algorithm used by GPT and most modern language models. By the end you will understand exactly how a raw string of text becomes a sequence of integer IDs ready to feed into an embedding table.

11.2 Why Not Just Use Characters?

The simplest possible tokenizer treats every character as its own token. The letter ‘a’ gets one ID, ‘b’ gets another, a space gets another, and so on. If you stick to ASCII you have 128 possible tokens, or 256 if you include the extended set. The vocabulary is tiny, you never encounter a character you have not seen before, and the implementation is trivial: the token ID for any character is just its ASCII code.

The problem is sequence length. The word “embedding” is 9 characters, so it becomes a sequence of 9 tokens. A sentence of 50 words might be 300 characters, which means the network has to process a sequence of 300 tokens and learn relationships across all of them. The longer the sequence, the harder it is for the network to connect information from the beginning to the end, especially for recurrent networks where the signal has to survive hundreds of time steps. A word-level tokenizer would represent that same sentence as roughly 50 tokens, which is a much shorter sequence for the network to handle.

The other issue is that individual characters carry almost no meaning on their own. The letter ‘e’ by itself tells the network nothing useful. The network has to learn to assemble characters into meaningful groups before it can start learning anything about language, which is extra work that a smarter tokenization scheme could avoid by handing the network pieces that already carry some meaning.

Figure 11-1. The same sentence at three granularities

Figure 11-1 shows the trade being made. Characters give a vocabulary small enough to fit in a byte and sequences long enough to drown a network. Words give short sequences and a vocabulary that can never be finished, since the first name or typo or compound noun the model has not seen becomes an unknown token and the meaning is simply lost. Subwords sit between the two, and the rest of this chapter is about how a tokenizer decides where to cut.

/* 061_Tokenizier.c */
#include <stdio.h>
#include <string.h>

int main(void)
{
    const char *text = "hello world";
    int i, len = strlen(text);

    printf("Text: \"%s\"\n\n", text);
    printf("Character tokens:\n");
    for (i = 0; i < len; i++)
        printf("  '%c' -> %d\n", text[i],
            (unsigned char)text[i]);

    printf("\nToken IDs: [");
    for (i = 0; i < len; i++)
        printf("%d%s", (unsigned char)text[i],
            i < len - 1 ? ", " : "");
    printf("]\n");

    printf("\nVocabulary size: 256 (one per byte)\n");
    printf("Sequence length: %d (one per "
           "character)\n", len);

    return 0;
}
Figure 11-2. A sentence split into one token per character

Figure 11-2 splits a sentence into one token per character. Character tokenization works, but the sequences it produces are long. The word “hello” alone becomes 5 tokens. A typical sentence runs 50 to 100 tokens. A full paragraph can be several hundred. Every additional token in the sequence is another step that the network has to process, and for recurrent networks each step depends on everything that came before it. The longer the sequence, the harder it is for gradients to flow all the way back to the beginning during training, and the more likely it is that information from early in the sequence gets diluted or lost by the time the network reaches the end. We saw this problem briefly with vanishing gradients in MLPs, and it gets much worse with sequences. We want a tokenization scheme that produces shorter sequences without losing information, which means each token needs to carry more meaning than a single character.

11.3 Word Tokenization

The opposite extreme is to treat every word as its own token. You split the text on spaces and punctuation, build a vocabulary of every unique word you encounter, and assign each one an integer ID. The word “hello” is now a single token instead of five characters, and a sentence of 20 words becomes a sequence of 20 tokens instead of 100 or more.

The problem moves to the other end of the scale. English has hundreds of thousands of words, and that is before you count misspellings, technical jargon, proper nouns, and words from other languages that show up in real text. A word-level vocabulary for a large dataset can easily reach 500,000 or more entries, which means the embedding table has 500,000 rows. That is a lot of parameters, and most of those rows will rarely get updated during training because the corresponding words barely appear in the data. Worse, any word the tokenizer has not seen during training is completely invisible at inference time. If someone types “microcontroller” and that word never appeared in the training corpus, the tokenizer has no ID for it and has to either ignore it or replace it with a generic unknown token. Word tokenization trades short sequences for a massive, fragile vocabulary.

/* 062_Word_Tokenization.c */
#include <stdio.h>
#include <string.h>

#define MAX_WORDS 100
#define MAX_WORD_LEN 32

typedef struct {
    char words[MAX_WORDS][MAX_WORD_LEN];
    int count;
}
Vocab;

static int vocab_add(Vocab *v, const char *word)
{
    int i;
    /* Check if word already exists */
    for (i = 0; i < v->count; i++)
        if (strcmp(v->words[i], word) == 0)
            return i;
    /* Add new word */
    strncpy(v->words[v->count], word, MAX_WORD_LEN - 1);
    v->words[v->count][MAX_WORD_LEN - 1] = '\0';
    return v->count++;
}

static int vocab_lookup(const Vocab *v, 
    const char *word)
{
    int i;
    for (i = 0; i < v->count; i++)
        if (strcmp(v->words[i], word) == 0)
            return i;
    return -1;  /* unknown word */
}

int main(void)
{
    const char
        *text = "the cat sat on the mat "
                "the dog sat on the rug";
    Vocab v = { .count = 0 };
    char buf[MAX_WORD_LEN];
    int tokens[MAX_WORDS];
    int n_tokens = 0;
    int i = 0, len = strlen(text);

    /* Tokenize by splitting on spaces */
    int start = 0;
    for (i = 0; i <= len; i++) {
        if (i == len || text[i] == ' ') {
            int wlen = i - start;
            if (wlen > 0 && wlen < MAX_WORD_LEN) {
                strncpy(buf, text + start, wlen);
                buf[wlen] = '\0';
                tokens[n_tokens++] = vocab_add(&v, buf);
            }
            start = i + 1;
        }
    }

    printf("Text: \"%s\"\n\n", text);
    printf("Vocabulary (%d words):\n", v.count);
    for (i = 0; i < v.count; i++)
        printf("  %d: \"%s\"\n", i, v.words[i]);

    printf("\nToken IDs: [");
    for (i = 0; i < n_tokens; i++)
        printf("%d%s", tokens[i],
            i < n_tokens - 1 ? ", " : "");
    printf("]\n");

    printf("\nSequence length: %d (one per "
           "word)\n", n_tokens);

    return 0;
}
Figure 11-3. The same sentence split into words

Figure 11-3 splits the same sentence into words instead. The sequence is shorter now, 12 tokens instead of the 46 characters that a character tokenizer would produce. But the vocabulary problem is immediate. Even in this tiny example there are 8 unique words, and we only processed one sentence. English has roughly 170,000 words in current use, and real text adds misspellings, technical terms, abbreviations, proper nouns, slang, and borrowed words from other languages. A realistic word vocabulary can easily reach into the hundreds of thousands or millions. An embedding table for a million words at 512 dimensions would need 2 billion floats, which is around 2 GB of memory just for the front door of the network. And any word that did not appear in the training data gets mapped to an unknown token, which means the network has no representation for it at all.

Neither extreme works well on its own. Character tokenization gives you a tiny vocabulary of 256 entries but produces sequences so long that the network struggles to connect information across them. Word tokenization gives you short, manageable sequences but requires an enormous vocabulary that wastes memory, trains slowly because most rows rarely get updated, and breaks completely on words it has never seen. What we need is something in between, a tokenization scheme that breaks text into pieces larger than characters but smaller than whole words, keeping the vocabulary at a reasonable size while producing sequences short enough for the network to handle.

11.4 The Idea Behind BPE

Byte Pair Encoding starts with the smallest possible tokens, individual characters, and builds up from there. The algorithm looks at the training text, finds the pair of adjacent tokens that appears most frequently, and merges that pair into a single new token. Then it repeats: find the most frequent pair in the updated sequence, merge it, and keep going. After enough rounds of merging, common words and word fragments become single tokens, while rare words stay broken into smaller pieces that the tokenizer can still represent.

The starting vocabulary is every unique character that appears in the training text, plus a special end-of-word marker that tells the tokenizer where word boundaries are. From there, each merge adds one new entry to the vocabulary. If the characters ‘t’ and ‘h’ appear next to each other more than any other pair, they get merged into a new token ‘th’. On the next round, maybe ‘th’ and ‘e’ are the most frequent pair, so they merge into ‘the’. After a few hundred or thousand merges, the vocabulary contains a mix of individual characters, common subword fragments like ‘ing’, ‘tion’, and ‘pre’, and frequent whole words like ‘the’ and ‘and’. The vocabulary size is a parameter you choose, and it directly controls the tradeoff between sequence length and vocabulary size.

Let us trace through the algorithm by hand first, then implement it.

Figure 11-4. Two merge steps on a four word corpus

Figure 11-4 follows the first two merges on the corpus the trace program uses. Everything starts as single characters with an end-of-word marker, and the pair e followed by s turns out to appear twice, in newest and in widest, so it becomes the single token es. Recounting after that merge finds es followed by t appearing twice as well, and the two collapse into est.

Nothing about that process knows what a word is. It found the suffix est because est happens to be the letter sequence that repeats, and the same procedure run on a corpus of C source would discover int and return for the same reason. That is why the vocabulary ends up matched to whatever the training text actually contains.

/* 063_BPE_Trace.c */
#include <stdio.h>
#include <string.h>

int main(void)
{
    printf("BPE algorithm trace:\n\n");
    printf("Training corpus: \"low lower newest "
           "widest\"\n\n");

    printf("Step 0 - Split into characters (with "
           "end-of-word marker '_'):\n");
    printf("  low:    l o w _\n");
    printf("  lower:  l o w e r _\n");
    printf("  newest: n e w e s t _\n");
    printf("  widest: w i d e s t _\n\n");

    printf("Step 1 - Count adjacent pairs:\n");
    printf("  (l,o): 2   (o,w): 2   (w,_): 1   (w,e): "
           "1\n");
    printf("  (e,r): 1   (r,_): 1   (n,e): 1   (e,w): "
           "1\n");
    printf("  (e,s): 2   (s,t): 2   (t,_): 2   (w,i): "
           "1\n");
    printf("  (i,d): 1   (d,e): 1\n\n");

    printf("Most frequent pairs: (l,o)=2, (o,w)=2, "
           "(e,s)=2, (s,t)=2, (t,_)=2\n");
    printf("Pick one (e.g., (e,s)) and merge into new "
           "token 'es'.\n\n");

    printf("Step 2 - After merging (e,s) -> 'es':\n");
    printf("  low:    l o w _\n");
    printf("  lower:  l o w e r _\n");
    printf("  newest: n e w es t _\n");
    printf("  widest: w i d es t _\n\n");

    printf("Step 3 - Recount pairs, merge most "
           "frequent...\n");
    printf("  (es,t): 2 -> merge into 'est'\n\n");

    printf("Step 4 - After merging (es,t) -> 'est':\n");
    printf("  low:    l o w _\n");
    printf("  lower:  l o w e r _\n");
    printf("  newest: n e w est _\n");
    printf("  widest: w i d est _\n\n");

    printf("Continue until desired vocabulary size is "
           "reached.\n");
    printf("Common subwords like 'est' become single "
           "tokens.\n");
    printf("Rare words are still decomposed into "
           "pieces.\n");

    return 0;
}
Figure 11-5. A hand trace of BPE on a four word corpus, pair counts and all

Figure 11-5 is a hand trace of BPE on a four word corpus, pair counts and all. The thing to take from it is that BPE builds up a vocabulary from the bottom. It starts with characters (guaranteed to cover everything) and learns which character sequences are common enough to deserve their own token. Common words like “the” become single tokens early. Less common words like “newest” stay split as “new” + “est” or similar. Completely unknown words can always be broken down to characters. No word is ever truly unknown.

11.5 Implementing BPE Training

Now let us implement the actual BPE training algorithm in C. We will use a small corpus so you can follow every merge step in the output, and we will run a handful of merge rounds so you can see the vocabulary grow from individual characters into subword tokens.

/* 064_BPE_Training.c */
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define MAX_TOKENS 200
#define MAX_TOKEN_LEN 16
#define MAX_WORDS 20
#define MAX_WORD_TOKENS 20

/* A word split into tokens */
typedef struct {
    char tokens[MAX_WORD_TOKENS][MAX_TOKEN_LEN];
    int n_tokens;
    int freq;   /* how many times this word appears */
}
Word;

/* Find the most frequent adjacent pair across all
   words */
static int find_best_pair(Word *words, int n_words, 
                           char best_a[MAX_TOKEN_LEN], 
                           char best_b[MAX_TOKEN_LEN])
{
    int best_count = 0;
    int w, t;

    for (w = 0; w < n_words; w++) {
        for (t = 0; t < words[w].n_tokens - 1; t++) {
            /* Count this pair across all words */
            int count = 0;
            int w2, t2;
            for (w2 = 0; w2 < n_words; w2++) {
                for (t2 = 0;
                 t2 < words[w2].n_tokens - 1;
                 t2++) {
                    if (strcmp(words[w2].tokens[t2],
                         words[w].tokens[t]) == 0 &&
                        strcmp(words[w2].tokens[t2+1], 
                            words[w].tokens[t+1]) == 0)
                        count += words[w2].freq;
                }
            }
            if (count > best_count) {
                best_count = count;
                strcpy(best_a, words[w].tokens[t]);
                strcpy(best_b, words[w].tokens[t+1]);
            }
        }
    }
    return best_count;
}

/* Merge all occurrences of (a, b) into "ab" */
static void merge_pair(Word *words, int n_words, 
                        const char *a, const char *b)
{
    char merged[MAX_TOKEN_LEN];
    snprintf(merged, MAX_TOKEN_LEN, "%s%s", a, b);

    int w, t;
    for (w = 0; w < n_words; w++) {
        for (t = 0; t < words[w].n_tokens - 1; t++) {
            if (strcmp(words[w].tokens[t], a) == 0 &&
                strcmp(words[w].tokens[t+1], b) == 0) {
                /* Replace token[t] with merged,
                   remove token[t+1] */
                strcpy(words[w].tokens[t], merged);
                int j;
                for (j = t + 1;
                     j < words[w].n_tokens - 1; j++)
                    strcpy(words[w].tokens[j], 
                        words[w].tokens[j+1]);
                words[w].n_tokens--;
                /* Don't advance t: check if the new
                   merged token
                   can merge with the next one too */
            }
        }
    }
}

static void print_words(const Word *words, int n_words)
{
    int w, t;
    for (w = 0; w < n_words; w++) {
        printf("  (%dx) ", words[w].freq);
        for (t = 0; t < words[w].n_tokens; t++)
            printf("[%s]", words[w].tokens[t]);
        printf("\n");
    }
}

int main(void)
{
    /* Corpus: word frequencies */
    const char *raw[] = { "low",  "lower",  "newest",
        "widest", "new" };
    int freqs[] = { 5, 2, 6, 3, 2 };
    int n_words = 5;

    Word words[MAX_WORDS];
    int w, i;

    /* Initialize: split each word into characters +
       end marker */
    for (w = 0; w < n_words; w++) {
        words[w].freq = freqs[w];
        words[w].n_tokens = 0;
        for (i = 0; raw[w][i]; i++) {
            char c[2] = { raw[w][i], '\0' };
            strcpy(words[w].tokens[words[w].n_tokens++],
                   c);
        }
        strcpy(words[w].tokens[words[w].n_tokens++],
               "_");
    }

    printf("BPE Training\n\n");
    printf("Initial state (characters + end "
           "marker):\n");
    print_words(words, n_words);

    /* Perform merge steps */
    int step;
    for (step = 0; step < 10; step++) {
        char a[MAX_TOKEN_LEN], b[MAX_TOKEN_LEN];
        int count = find_best_pair(words, 
            n_words, a, b);
        if (count < 2) break;
        /* stop when no pair appears 2+ times */

        printf("\nMerge %d: (%s, %s) count=%d -> "
               "[%s%s]\n",
               step + 1, a, b, count, a, b);
        merge_pair(words, n_words, a, b);
        print_words(words, n_words);
    }

    return 0;
}
Figure 11-6. BPE learning its merge rules in order

Figure 11-6 has BPE learning its merge rules in order, most frequent pair first. The merge rules are applied in the order they were learned, which is the only order that reproduces the training vocabulary. Reading the output, newest becomes new, est and the word boundary marker, three tokens where there were seven characters, and lower keeps its low prefix but has no rule covering er so those characters survive on their own.

The last row is the one worth dwelling on. Nothing in the training corpus resembles unknown, so no merge rule fires and the word falls all the way back to eight single characters. That is not a failure case being handled gracefully, it is the ordinary behavior of the scheme, and it is why a BPE tokenizer never needs an unknown token at all.

11.6 Applying BPE to New Text

Once the merge rules are learned, tokenizing new text means: split into characters, then apply each merge rule in order.

/* 065_BPE_New_Text.c */
#include <stdio.h>
#include <string.h>

#define MAX_TOKENS 50
#define MAX_TOKEN_LEN 16
#define MAX_RULES 20

typedef struct {
    char a[MAX_TOKEN_LEN];
    char b[MAX_TOKEN_LEN];
    char merged[MAX_TOKEN_LEN];
}
MergeRule;

static void apply_merges(
        char tokens[MAX_TOKENS][MAX_TOKEN_LEN],
                          int *n_tokens, 
                          const MergeRule
                              *rules, int n_rules)
{
    int r, t;
    for (r = 0; r < n_rules; r++) {
        for (t = 0; t < *n_tokens - 1; t++) {
            if (strcmp(tokens[t], rules[r].a) == 0 &&
                strcmp(tokens[t+1], rules[r].b) == 0) {
                strcpy(tokens[t], rules[r].merged);
                int j;
                for (j = t + 1; j < *n_tokens - 1; j++)
                    strcpy(tokens[j], tokens[j+1]);
                (*n_tokens)--;
                t--;  /* recheck at same position */
            }
        }
    }
}

/* Simple token-to-ID mapping */
static int token_to_id(const char *token, 
                        char vocab[100][MAX_TOKEN_LEN], 
                            int *vocab_size)
{
    int i;
    for (i = 0; i < *vocab_size; i++)
        if (strcmp(vocab[i], token) == 0)
            return i;
    strcpy(vocab[*vocab_size], token);
    return (*vocab_size)++;
}

int main(void)
{
    /* Pretend these merge rules were learned from
       training */
    MergeRule rules[] = {
        { "e", "s", "es" },
        { "es", "t", "est" },
        { "l", "o", "lo" },
        { "lo", "w", "low" },
        { "n", "e", "ne" },
        { "ne", "w", "new" },
    };
    int n_rules = 6;

    /* Tokenize some text */
    const char *test_words[] = { "newest", "lower",
        "low", "newest", "unknown" };
    int n_test = 5;

    char vocab[100][MAX_TOKEN_LEN];
    int vocab_size = 0;
    int w, i;

    printf("Merge rules (learned from training):\n");
    for (i = 0; i < n_rules; i++)
        printf("  %s + %s -> %s\n", rules[i].a,
            rules[i].b, rules[i].merged);

    printf("\nTokenizing new text:\n\n");

    for (w = 0; w < n_test; w++) {
        char tokens[MAX_TOKENS][MAX_TOKEN_LEN];
        int n_tokens = 0;

        /* Split into characters + end marker */
        for (i = 0; test_words[w][i]; i++) {
            char c[2] = { test_words[w][i], '\0' };
            strcpy(tokens[n_tokens++], c);
        }
        strcpy(tokens[n_tokens++], "_");

        /* Apply merge rules */
        apply_merges(tokens, &n_tokens, rules, n_rules);

        /* Print result */
        printf("  \"%s\" -> ", test_words[w]);
        for (i = 0; i < n_tokens; i++)
            printf("[%s]", tokens[i]);

        /* Assign IDs */
        printf("  IDs: [");
        for (i = 0; i < n_tokens; i++) {
            int id = token_to_id(tokens[i], vocab, 
                &vocab_size);
            printf("%d%s", id,
                i < n_tokens - 1 ? ", " : "");
        }
        printf("]\n");
    }

    printf("\nVocabulary (%d tokens):\n", vocab_size);
    for (i = 0; i < vocab_size; i++)
        printf("  %d: \"%s\"\n", i, vocab[i]);

    printf("\n\"unknown\" was split into characters "
           "because no merge\n");
    printf("rules matched. BPE can handle any "
           "input.\n");

    return 0;
}
Figure 11-7. The learned merges applied to unseen text

Figure 11-7 applies the learned merges to unseen text, and one word falls all the way back to characters. Watch the merges happen in the output. Each round, the algorithm scans every adjacent pair of tokens across all words, weighted by how often each word appears in the corpus. The pair with the highest total count gets merged into a single new token. After several rounds, common subword fragments like “est”, “low”, and “new” emerge as single tokens, while rarer parts of words stay broken into smaller pieces.

The sequence of merge rules, which pairs were merged and in what order, is the entire definition of the tokenizer. To tokenize new text at inference time, you do not retrain anything. You take the input string, split it into individual characters, and then replay the same merge rules in the same order they were learned during training. If the first rule says merge ‘e’ and ‘s’ into ‘es’, you scan the character sequence and merge every adjacent (‘e’, ‘s’) pair. Then you apply the second rule, then the third, and so on. By the end, common words and subwords have been collapsed into single tokens, and anything the tokenizer has never seen gets left as smaller pieces that the vocabulary can still represent. Nothing is ever truly unknown, because at worst a word stays as individual characters that are always in the vocabulary.

11.7 The Full Pipeline

Let us trace the complete pipeline from raw text to the input of a neural network.

/* 066_Full_Pipeline.c */
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define EMBED_DIM 4
#define MAX_SEQ 20

int main(void)
{
    /* Simulated vocabulary (in a real system, this
       comes from BPE training) */
    const char *vocab[] = {
        "the", "cat", "sat", "on", "mat", "a",
            "big", "red",
        "dog", "ran", "fast", "<UNK>"
    };
    int vocab_size = 12;

    /* Simulated embeddings (in a real system, these
       are learned) */
    float embeddings[12][EMBED_DIM];
    srand(42);
    int i, j;
    for (i = 0; i < vocab_size; i++)
        for (j = 0; j < EMBED_DIM; j++)
            embeddings[i][j] =
                ((float)rand() / RAND_MAX) * 2 - 1;

    /* Input text */
    const char *text = "the big red cat sat on a mat";

    /* Step 1: Tokenize (simple word split for this
       demo) */
    int token_ids[MAX_SEQ];
    int seq_len = 0;
    char buf[32];
    int start = 0, len = strlen(text);

    printf("=== Full Pipeline: Text to Vectors "
           "===\n\n");
    printf("Step 1: Tokenize\n");
    printf("  Input: \"%s\"\n  Tokens: ", text);

    for (i = 0; i <= len; i++) {
        if (i == len || text[i] == ' ') {
            int wlen = i - start;
            if (wlen > 0 && wlen < 32) {
                strncpy(buf, text + start, wlen);
                buf[wlen] = '\0';

                /* Look up in vocab */
                int id = vocab_size - 1;  /* <UNK> */
                for (j = 0; j < vocab_size; j++) {
                    if (strcmp(vocab[j], buf) == 0) {
                        id = j;
                        break;
                    }
                }
                token_ids[seq_len++] = id;
                printf("\"%s\"(%d) ", buf, id);
            }
            start = i + 1;
        }
    }

    /* Step 2: Token IDs */
    printf("\n\nStep 2: Token IDs\n  [");
    for (i = 0; i < seq_len; i++)
        printf("%d%s", token_ids[i],
            i < seq_len - 1 ? ", " : "");
    printf("]\n");

    /* Step 3: Embedding lookup */
    printf("\nStep 3: Embedding Lookup "
           "(%d-dimensional)\n", EMBED_DIM);
    for (i = 0; i < seq_len; i++) {
        int id = token_ids[i];
        printf("  \"%s\" (id=%d) -> [", vocab[id], id);
        for (j = 0; j < EMBED_DIM; j++)
            printf("%+.2f%s", embeddings[id][j],
                j < EMBED_DIM - 1 ? ", " : "");
        printf("]\n");
    }

    printf("\nStep 4: Feed sequence of %d vectors into "
           "neural network\n", seq_len);
    printf("  Each vector is "
           "%d-dimensional\n", EMBED_DIM);
    printf("  Input tensor shape: [%d, %d]\n",
        seq_len, EMBED_DIM);
    printf("\n  This is where RNNs, LSTMs, or "
           "Transformers take over.\n");

    return 0;
}
Figure 11-8. Raw text to tokens to IDs to vectors, all four stages

Figure 11-8 runs raw text to tokens to IDs to vectors, all four stages. The full pipeline is now visible end to end. The raw string “the big red cat sat on a mat” gets split into 8 words, each word gets mapped to an integer ID through a vocabulary lookup, and each ID gets replaced with a 4-dimensional vector from the embedding table. The result is a matrix with shape [8, 4], eight rows of four floats each, where every row is the learned representation of one token.

This matrix is what the rest of the network actually sees. It never touches raw text. It never sees token IDs. Everything downstream operates on this sequence of continuous vectors, and the quality of those vectors determines how much the network has to work with. At this point the embeddings are random, so the vectors carry no meaning. After training, each row would encode something about the word it represents, its typical context, its grammatical role, its relationship to other words, all compressed into four numbers. In a real system the embedding dimension would be much larger, 256 or 512 or more, to give each token enough room to encode the information the network needs.

This matrix is the input to whatever sequence processing architecture comes next. In Chapter 13 we will feed it into a recurrent neural network that processes one vector at a time. In Chapter 15 we will use an LSTM that handles longer sequences. And in Chapter 25 we will build a transformer that processes all the vectors in parallel using attention. The architectures are very different, but they all start from the same place: a sequence of learned embedding vectors.

11.8 WordPiece vs BPE

WordPiece, which is the tokenization algorithm used by BERT, works on the same principle as BPE but makes its merge decisions differently. BPE simply counts which pair of adjacent tokens appears most often and merges that pair. WordPiece is more careful: it evaluates each candidate merge by asking how much the merge would improve the likelihood of the training data, which means it considers not just how often a pair appears but how informative the merged token would be relative to its individual pieces. A pair that appears frequently but whose components are also common on their own might not get merged as quickly as a less frequent pair whose components rarely appear apart.

Figure 11-9. BPE and WordPiece compared

Figure 11-9 sets BPE and WordPiece against each other. In practice, the difference is subtle. Both algorithms produce subword vocabularies in the 30,000 to 50,000 range. Both can handle any input text without producing unknown tokens, because both fall back to individual characters for anything the merge rules do not cover. Both strike a similar balance between sequence length and vocabulary size. The main split is along architectural lines: BPE is the standard for GPT, LLaMA, and most decoder-only language models, while WordPiece is used by BERT and its encoder-based variants. For the models we build in this book, we will use BPE.

11.9 Key Takeaways

11.10 Exercises