Decoder-Only Transformer

GPT-style architecture, the foundation of modern LLMs

26.1 What You Will Learn

Chapter 25 assembled the encoder-decoder transformer and finished by noting what it costs. At the original dimensions a six and six arrangement spends 18.9M on the encoder against 25.2M on the decoder, so roughly a third of the parameter budget exists to read an input that the decoder is perfectly capable of reading itself. That observation is the beginning of this chapter, because the architecture behind GPT, LLaMA, DeepSeek-V3 and most other current language model is what you get by acting on it. Throw away the encoder, throw away cross-attention, keep the causal mask, and run a single stack over a single sequence.

The simplification is larger than it sounds. With one stack there is no longer a distinction between input and output, only a sequence and a rule that says each position predicts the one after it. The prompt a user types and the text the model generates are the same object, handled by the same code, and the boundary between them exists only in the mind of whoever typed the prompt. That collapse is why decoder-only models are easy to scale and easy to reuse, since anything expressible as text becomes a training example without a schema for what is input and what is output.

This chapter builds one. Section 26.3 sets up next-token prediction and shows how much training signal a single sequence yields. Section 26.4 verifies the causal mask by measurement rather than assertion, which turns out to be worth doing because the property is easy to break and impossible to notice by eye. Section 26.5 assembles a complete small GPT with weight tying, Section 26.6 generates from it one token at a time, and both sections turn up the same instructive failure that an untrained model with tied weights produces.

26.2 Why Decoder-Only

The encoder-decoder split earns its keep when the input and the output are genuinely different objects. Translating English to French involves two sequences, in two languages, of two different lengths and two different vocabularies, and giving each its own stack with its own weights is a reasonable response to a genuine asymmetry. Summarization has the same shape with a long input and a short output, and Chapter 25 noted that the encoder is at its cheapest exactly there, since it runs once over the source regardless of how many output tokens eventually follow. Nothing in this chapter argues that the arrangement was a mistake, only that language modelling is not the task it was built for.

Language modelling has no such split to exploit. The task is to read “The cat sat on the” and produce “mat”, and there is exactly one sequence involved from beginning to end. The thing being read and the thing being written are the same text at two different lengths, which reduces a separate encoder to a stack that reads a prefix the decoder is already reading with weights the decoder already has. Deleting it costs nothing in capability, saves roughly a third of the parameters at matched depth, and removes one sublayer from every block that remains, since cross-attention has nothing left to attend to.

What replaces the split is a convention. At position i the model predicts the token at position i+1, using only positions 0 through i, and the causal mask is what enforces the restriction. That single rule turns any text at all into training data, with no need to label which part is input and which is output, which is the property that let the field train on the open internet rather than on curated pairs. An encoder-decoder model needs matched examples, an English sentence with its French translation attached, and those are expensive and finite. A decoder-only model needs text.

The last argument is architectural rather than practical. A decoder-only stack is a single repeated unit, identical from the first layer to the last, which means scaling it is a matter of changing three numbers. Encoder-decoder models have two different block types, a connection between the stacks, and a choice about how deep to make each half relative to the other. When the field started training models measured in hundreds of billions of parameters, the architecture with fewer decisions in it won.

26.3 Next-Token Prediction

Before any model, the data arrangement, because it is the part people skip and the part that explains why this architecture won. A sequence of tokens becomes a set of prediction tasks by pairing each position with the one immediately after it, so the same array serves as both the inputs and the targets with nothing more than an offset separating the two roles. There is no separate label file, no alignment step, and no decision about what counts as a question and what counts as an answer. The program below prints those pairs for an eight token sequence so the shape is unmistakable before any weights get involved.

/* 135_Next_Token.c */
#include <stdio.h>

int main(void)
{
    /* A simple token sequence */
    int tokens[] = { 4, 2, 7, 1, 5, 3, 6, 0 };
    int len = 8;
    int i;

    printf("Next-token prediction setup:\n\n");
    printf("  Tokens:  ");
    for (i = 0; i < len; i++) printf("%d ", tokens[i]);
    printf("\n\n");

    printf("  Position  Input  Target\n");
    printf("  --------  -----  ------\n");
    for (i = 0; i < len - 1; i++) {
        printf("  %4d      %3d    %3d\n", i,
               tokens[i], tokens[i + 1]);
    }

    printf("\n  At each position the model sees\n");
    printf("  tokens[0..i] and predicts "
           "tokens[i+1].\n");
    printf("  The causal mask is what stops it\n");
    printf("  from seeing future tokens.\n\n");

    printf("  In training, ALL positions "
           "are trained\n");
    printf("  at once.\n");
    printf("  The loss sums cross-entropy "
           "over them.\n");
    printf("  Seq2seq trains only on the output\n");
    printf("  side, so this gets far more from\n");
    printf("  the same text.\n");

    return 0;
}

Eight tokens produce seven rows, because the last token has nothing after it to predict. Position 0 sees token 4 and must produce token 2, position 1 sees token 2 and must produce token 7, and so on down to position 6. The input column is simply the token array and the target column is the same array shifted left by one, which is all that next-token prediction means at the data level.

The consequence stated at the bottom of the output is the one that matters. Every position carries a training signal, so a sequence of a thousand tokens yields 999 supervised examples rather than one. Compare that against the sequence to sequence arrangement of Chapter 18, where a source sentence and its translation gave a training signal only on the target side, so half the tokens the model read contributed nothing directly to the loss. A decoder-only model extracts a gradient from every token it is shown.

That efficiency is only available because of the causal mask, and the connection is worth making explicit. All 999 predictions are computed in a single forward pass over the whole sequence, with position 3 predicting token 4 while position 700 predicts token 701, simultaneously and without interference. The mask is what prevents position 3 from seeing token 4 while it does so. Remove the mask and every position could read its own answer, which would give a model that scores perfectly during training and produces nothing during generation, since at generation time the answer genuinely is not there.

Note also what this means for the loss and for how the field talks about scale. There is one cross-entropy term per position and the total is their sum, so a sequence of two thousand tokens contributes roughly twice the gradient of a sequence of one thousand, and the quantity that determines how much a training step actually learns is the token count in the batch rather than the number of sequences in it. That is why model training budgets are quoted in tokens, why context length and batch size trade against each other on a fixed hardware budget, and why the scaling laws of Chapter 32 are written in terms of tokens rather than examples.

26.4 The Decoder-Only Block

The block itself needs no new code, which is the point of having built the pieces separately. It is the transformer block of Chapter 24, with the causal mask of Chapter 25 applied inside its self-attention, and with the cross-attention sublayer of Chapter 25 deleted, which returns it to the two sublayers Chapter 24 started with. Everything else is unchanged, including the pre-norm placement, the residual additions and the widening feedforward network, so a reader who followed Chapter 24 already knows how this block behaves apart from the one restriction the mask imposes.

That leaves a question worth answering with a measurement. The mask is one comparison and one assignment buried inside a scoring loop, it produces no error if it is wrong, and a broken mask yields a model that trains beautifully and generates gibberish. So rather than print the block’s output and assert that it is causal, this program tests the property directly. It runs the block twice on the same sequence, changing only the very last token between runs, and reports how far each position’s output moved.

/* 136_Decoder_Only_Block.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#include <stdlib.h>

static float randf(void)
{
    return (float)rand() / RAND_MAX;
}
static float gelu(float x)
{
    float c = 0.7978846f;
    float u = x + 0.044715f * x * x * x;
    return 0.5f * x * (1.0f + tanhf(c * u));
}
static float dot(const float *a, const float *b, 
                 int n)
{
    float s = 0;
    int i;
    for (i = 0; i < n; i++) s += a[i] * b[i];
    return s;
}
static void softmax(float *x, int n)
{
    float mx = -FLT_MAX, s = 0;
    int i;
    for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
    for (i = 0; i < n; i++) {
        x[i] = expf(x[i] - mx);
        s += x[i];
    }
    for (i = 0; i < n; i++) x[i] /= s;
}
static void rmsnorm(const float *x, float *out, 
                    int n)
{
    int i;
    double ss = 0;
    float eps = 1e-5f;
    for (i = 0; i < n; i++) ss += x[i] * x[i];
    float ri = 1.0f / sqrtf((float)(ss/n) + eps);
    for (i = 0; i < n; i++) out[i] = x[i] * ri;
}

static void init_random(float *p, int n, float scale)
{
    int i;
    for (i = 0; i < n; i++)
        p[i] = (randf() * 2 - 1) * scale;
}

#define SL 5
#define DM 8
#define NH 2
#define DH (DM/NH)
#define DF (DM*4)

typedef struct {
    float WQ[NH][DH][DM], WK[NH][DH][DM];
    float WV[NH][DH][DM], WO[DM][DM];
    float W1[DF][DM], W2[DM][DF];
}
Block;

static void block_forward(const Block *b, 
    float X[SL][DM])
{
    float norm[SL][DM], sub[SL][DM];
    int h, i, j, k, d;

    /* Causal self-attention */
    for (i = 0; i < SL; i++) rmsnorm(X[i], norm[i], DM);

    float ho[NH][SL][DH];
    float scale = 1.0f / sqrtf((float)DH);
    for (h = 0; h < NH; h++)
        for (i = 0; i < SL; i++) {
            float q[DH];
            for (k = 0; k < DH; k++) {
                q[k] = 0;
                for (d = 0; d < DM; d++)
                    q[k] += b->WQ[h][k][d] * norm[i][d];
            }
            float sc[SL];
            for (j = 0; j < SL; j++) {
                float key[DH];
                for (k = 0; k < DH; k++) {
                    key[k] = 0;
                    for (d = 0; d < DM; d++)
                        key[k] += b->WK[h][k][d]
                                  * norm[j][d];
                }
                sc[j] = dot(q, key, DH) * scale;
                if (j > i) sc[j] = -1e9f;
                /* CAUSAL MASK */
            }
            softmax(sc, SL);
            for (k = 0; k < DH; k++) {
                ho[h][i][k] = 0;
                for (j = 0; j < SL; j++) {
                    float v = 0;
                    for (d = 0; d < DM; d++)
                        v += b
                            ->WV[h][k][d] * norm[j][d];
                    ho[h][i][k] += sc[j] * v;
                }
            }
        }
    for (i = 0; i < SL; i++) {
        float cat[DM];
        for (h = 0; h < NH; h++)
            for (k = 0; k < DH; k++)
                cat[h*DH+k] = ho[h][i][k];
        for (j = 0; j < DM; j++) {
            sub[i][j] = 0;
            for (k = 0; k < DM; k++)
                sub[i][j] += b->WO[j][k] * cat[k];
        }
    }
    for (i = 0; i < SL; i++)
        for (j = 0; j < DM; j++) X[i][j] += sub[i][j];

    /* FFN */
    for (i = 0; i < SL; i++) rmsnorm(X[i], norm[i], DM);
    for (i = 0; i < SL; i++) {
        float hid[DF];
        int f;
        for (f = 0; f < DF; f++) {
            float z = 0;
            for (j = 0; j < DM; j++)
                z += b->W1[f][j] * norm[i][j];
            hid[f] = gelu(z);
        }
        for (j = 0; j < DM; j++) {
            float z = 0;
            for (f = 0; f < DF; f++)
                z += b->W2[j][f] * hid[f];
            X[i][j] += z;
        }
    }
}

int main(void)
{
    Block blk;
    float X[SL][DM], Y[SL][DM], Z[SL][DM];
    int i, j;

    srand(42);
    init_random((float*)&blk, 
        sizeof(Block)/sizeof(float), 
                0.1f);
    for (i = 0; i < SL; i++)
        for (j = 0; j < DM; j++)
            X[i][j] = randf() * 1.0f - 0.5f;

    /* Run 1, the sequence as given */
    for (i = 0; i < SL; i++)
        for (j = 0; j < DM; j++) Y[i][j] = X[i][j];
    block_forward(&blk, Y);

    /* Run 2, with the LAST token completely replaced */
    for (i = 0; i < SL; i++)
        for (j = 0; j < DM; j++) Z[i][j] = X[i][j];
    for (j = 0; j < DM; j++) Z[SL-1][j] = -Z[SL-1][j];
    block_forward(&blk, Z);

    printf("Causal block, changing only the last "
           "token\n\n");
    printf("  pos   |output|   change from run 1\n");
    printf("  ---   --------   -------------------\n");
    for (i = 0; i < SL; i++) {
        float n = 0, d = 0;
        for (j = 0; j < DM; j++) {
            n += Y[i][j] * Y[i][j];
            d += (Z[i][j] - Y[i][j])
                * (Z[i][j] - Y[i][j]);
        }
        printf("  %3d    %.5f    %.5f%s\n", i,
               sqrtf(n), sqrtf(d), 
               i == SL
                   -1 ? "   <- the changed one" : "");
    }

    printf("\nPositions 0 to %d did not move at all. "
           "The\n",
           SL - 2);
    printf("mask stops them from ever "
           "seeing position\n");
    printf("%d, so replacing it cannot reach them. "
           "Only\n",
           SL - 1);
    printf("the last position changed, "
           "and it changed\n");
    printf("because it is the one that "
           "was edited.\n\n");

    printf("Run the same test on an unmasked block "
           "and\n");
    printf("every row moves. That difference is the\n");
    printf("whole of what causal masking "
           "buys, and it\n");
    printf("is what lets one forward pass produce a\n");
    printf("training signal at every position at "
           "once.\n");

    return 0;
}

The result is exact rather than approximate, which is the useful part. Positions 0 through 3 report a change of 0.00000, and that is a genuine zero rather than a small number rounded down, since the arithmetic those positions perform never touches the token that was replaced. Position 4 reports 1.42223 against its own output norm of 0.70481, so it moved about twice its own length, which is what you expect from a token whose every component was negated. Replacing the final token of the sequence had no effect whatsoever on anything before it.

That is the defining property of a causal model and it follows directly from the mask. Position 2′s attention scores against positions 3 and 4 are overwritten with a large negative number before the softmax runs, so those positions receive weight zero, so their values contribute nothing to position 2′s output. Change them however you like and position 2 cannot tell. Position 4 changed because position 4 is the token that was edited, and it would have changed under any masking scheme at all.

This test is worth keeping as a habit rather than a one-off. Attention code is dense with index arithmetic and it is entirely possible to write a mask that compares the wrong pair of indices, or applies after the softmax rather than before, or masks the transpose of the intended triangle. None of those mistakes produces a compiler warning or a runtime error, and the resulting model still trains, because a model that can see its own answer learns to copy it and reports a very low loss while doing so. Perturbing a late token and checking that early outputs are bit for bit unchanged catches every variant of the bug in a few lines.

The comparison in the closing text is the other half. An unmasked block, which is exactly what Chapter 24 built, would show every row moving, because bidirectional attention means position 0 reads position 4 as readily as it reads itself. Neither behaviour is wrong. They are the encoder and the decoder, and the only difference between them in code is the one line that sets a score to negative 1e9.

26.5 A Complete GPT

Everything assembles now, and the assembly is shorter than Chapter 25′s because there is only one stack to wire up. Token embeddings and positional encodings go in the front, a stack of decoder-only blocks runs over the result, a final normalization cleans up the residual stream that has been accumulating since the first block, and an output projection turns each position’s vector into a score for every token in the vocabulary. Chapter 24 noted that the final normalization exists because a pre-norm stack never rescales its residual stream directly and the magnitude drifts upward with depth, so something has to bring it back before the output head reads it.

Figure 26-3 is the whole model on one page, which is the first time in the book that has been possible. One stack, no cross-attention, no second input, and every part of it built in an earlier chapter. Compare it against Figure 25-1 and roughly half the diagram is simply absent.

The output projection is the one place this differs materially from Chapter 25. Rather than learning a fresh matrix of vocabulary size by d_model, which for a realistic vocabulary is one of the largest single objects in the model, the implementation reuses the embedding table transposed. The trick is called weight tying, and the same matrix that turns a token index into a vector on the way into the stack turns a vector back into a score per token on the way out. It saves the parameters and it also couples the two spaces, so a token whose embedding moves during training has its output behaviour move with it rather than the two drifting apart.

/* 137_Gpt.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#include <stdlib.h>
#include <string.h>

static float randf(void)
{
    return (float)rand() / RAND_MAX;
}
static float gelu(float x)
{
    float c = 0.7978846f;
    float u = x + 0.044715f * x * x * x;
    return 0.5f * x * (1.0f + tanhf(c * u));
}
static float dot(const float *a, const float *b, 
                 int n)
{
    float s = 0;
    int i;
    for (i = 0; i < n; i++) s += a[i] * b[i];
    return s;
}
static void softmax(float *x, int n)
{
    float mx = -FLT_MAX, s = 0;
    int i;
    for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
    for (i = 0; i < n; i++) {
        x[i] = expf(x[i] - mx);
        s += x[i];
    }
    for (i = 0; i < n; i++) x[i] /= s;
}
static void rmsnorm(const float *x, float *out, 
                    int n)
{
    int i;
    double ss = 0;
    float eps = 1e-5f;
    for (i = 0; i < n; i++) ss += x[i] * x[i];
    float ri = 1.0f / sqrtf((float)(ss/n) + eps);
    for (i = 0; i < n; i++) out[i] = x[i] * ri;
}

#define VOCAB 8
#define SL 6
#define DM 8
#define NH 2
#define DH (DM/NH)
#define DF 16
#define N_BLOCKS 2

typedef struct {
    float WQ[NH][DH][DM], WK[NH][DH][DM];
    float WV[NH][DH][DM], WO[DM][DM];
    float W1[DF][DM], W2[DM][DF];
}
Block;

static void block_fwd(const Block *b, 
    float X[][DM], int sl)
{
    float norm[SL][DM];
    int h, i, j, k, d;

    /* Causal self-attention */
    for (i = 0; i < sl; i++) rmsnorm(X[i], norm[i], DM);
    float ho[NH][SL][DH], scale = 1.0f/sqrtf((float)DH);
    for (h = 0;h<NH;h++) for (i = 0;i<sl;i++) {
        float q[DH];
        for (k = 0; k < DH; k++) {
            q[k] = 0;
            for (d = 0; d < DM; d++)
                q[k] += b->WQ[h][k][d] * norm[i][d];
        }
        float sc[SL];
        for (j = 0; j < sl; j++) {
            float key[DH];
            for (k = 0; k < DH; k++) {
                key[k] = 0;
                for (d = 0; d < DM; d++)
                    key[k] += b
                        ->WK[h][k][d] * norm[j][d];
            }
            sc[j] = dot(q, key, DH) * scale;
            /* the causal mask, one line */
            if (j > i) sc[j] = -1e9f;
        }
        softmax(sc, sl);
        for (k = 0; k < DH; k++) {
            ho[h][i][k] = 0;
            for (j = 0; j < sl; j++) {
                float v = 0;
                for (d = 0; d < DM; d++)
                    v += b->WV[h][k][d] * norm[j][d];
                ho[h][i][k] += sc[j] * v;
            }
        }
    }
    for (i = 0; i < sl; i++) {
        float cat[DM];
        for (h = 0; h < NH; h++)
            for (k = 0; k < DH; k++)
                cat[h*DH + k] = ho[h][i][k];
        float sub[DM];
        for (j = 0; j < DM; j++) {
            sub[j] = 0;
            for (k = 0; k < DM; k++)
                sub[j] += b->WO[j][k] * cat[k];
        }
        for(j = 0;j<DM;j++)X[i][j] += sub[j];
        }

    /* FFN */
    for(i = 0;i<sl;i++) rmsnorm(X[i], norm[i], DM);
    for(i = 0;i<sl;i++) {
        float hid[DF];
        int f;
        for (f = 0; f < DF; f++) {
            float z = 0;
            for (j = 0; j < DM; j++)
                z += b->W1[f][j] * norm[i][j];
            hid[f] = gelu(z);
        }
        for (j = 0; j < DM; j++) {
            float z = 0;
            for (int f = 0; f < DF; f++)
                z += b->W2[j][f] * hid[f];
            X[i][j] += z;
        }
    }
}

int main(void)
{
    /* Token embeddings */
    float embed[VOCAB][DM];
    /* Positional embeddings (learned) */
    float pos_embed[SL][DM];
    /* Transformer blocks */
    Block blocks[N_BLOCKS];
    /* Output head (weight-tied with embedding) */

    int i, j;
    srand(42);
    for (i = 0; i < VOCAB; i++)
        for (j = 0; j < DM; j++)
            embed[i][j] = (randf()*2-1) * 0.3f;
    for (i = 0; i < SL; i++)
        for (j = 0; j < DM; j++)
            pos_embed[i][j] = (randf()*2-1) * 0.1f;
    for (int b = 0; b < N_BLOCKS; b++) {
        float *w = (float*)&blocks[b];
        int n = (int)(sizeof(Block)/sizeof(float));
        for (i = 0; i < n; i++)
            w[i] = (randf()*2-1) * 0.1f;
    }

    /* Input sequence */
    int tokens[] = { 3, 1, 4, 1, 5, 2 };
    int seq_len = 6;

    printf("GPT-style Decoder-Only Transformer\n");
    printf("  vocab=%d  d_model=%d  heads=%d  "
           "layers=%d\n\n", VOCAB, DM, NH, N_BLOCKS);

    /* 1. Token embedding + position embedding */
    float X[SL][DM];
    for (i = 0; i < seq_len; i++)
        for (j = 0; j < DM; j++)
            X[i][j] = embed[tokens[i]][j]
                + pos_embed[i][j];

    printf("Input tokens: [");
    for (i = 0; i < seq_len; i++)
        printf("%d%s", tokens[i],
               i<seq_len-1?", ":"");
    printf("]\n\n");

    /* 2. Transformer blocks */
    for (int b = 0; b < N_BLOCKS; b++) {
        block_fwd(&blocks[b], X, seq_len);
        printf("  After block %d: "
               "pos0 norm=%.3f\n", b+1,
               sqrtf(dot(X[0], X[0], DM)));
    }

    /* 3. Final RMSNorm */
    for (i = 0; i < seq_len; i++)
        rmsnorm(X[i], X[i], DM);

    /* 4. Output head, weight-tied with the
          embedding table */
    printf("\nNext-token predictions:\n\n");
    printf("  Position  Context          Predicted  "
           "Actual\n");
    printf("  --------  ---------------  ---------  "
           "------\n");

    for (i = 0; i < seq_len - 1; i++) {
        /* Logits = X[i] . embed[v]^T (weight tying) */
        float logits[VOCAB];
        for (j = 0; j < VOCAB; j++)
            logits[j] = dot(X[i], embed[j], DM);
        softmax(logits, VOCAB);

        int pred = 0;
        for (j = 1; j < VOCAB; j++)
            if (logits[j] > logits[pred]) pred = j;

        printf("  %4d      tokens[0..%d]    %5d      "
               "%5d\n",
               i, i, pred, tokens[i+1]);
    }

    printf("\nWeight tying, the output projection\n");
    printf("reuses the embedding matrix transposed.\n");
    printf("It saves parameters, and works because\n");
    printf("similar tokens should have similar\n");
    printf("embeddings AND similar output logits.\n");

    /* Parameter count */
    int emb_params = VOCAB * DM + SL * DM;
    int block_params = 
        N_BLOCKS * (4*DM*DM + 2*DM*DF + 2*DM);
    /* the output head is tied, so it is free */
    int total = emb_params + block_params;
    printf("\nParameters: %d embedding + %d blocks "
           "= %d\n",
           emb_params, block_params, total);
    printf("  The tied output head adds 0.\n");

    return 0;
}

Read the prediction table against the input line above it and something jumps out immediately. The input tokens are 3, 1, 4, 1, 5, 2 and the predictions at positions 0 through 4 come out as 3, 1, 4, 1, 5, which is the input sequence again with its last element dropped. The model is not predicting the next token at all, it is reproducing the current one, exactly, at every position, including the repeated 1 at positions 1 and 3 where a coincidence would have been unlikely to hold. The Actual column beside it shows what the targets were, and the model gets every one of them wrong in the same systematic way.

That is not a bug, it is weight tying meeting an untrained network, and the mechanism is worth following because it explains the rest of the chapter. Section 24.5 established that a freshly initialised block sits close to the identity, since the residual passes its input through and both sublayers contribute almost nothing. Stack two of them and the vector arriving at the output head is still approximately the embedding of the input token. The output head then scores that vector against every embedding in the table by dot product, and a vector’s largest dot product is with itself. So the model reports the input token with the highest score, at every position, for a completely mechanical reason.

Two useful things follow. The first is that weight tying is doing exactly what it advertises even before training, since the geometry of the embedding space is already the geometry of the output space and the two cannot drift apart. That is the whole argument for tying, and it also saves a matrix, in this case 64 parameters against a total of 1168 but in GPT-2 Small it is 38.6M against 124M, roughly a third of the model. The second is that copying the input is the correct baseline behaviour to start training from, since the next token in real text is more often related to the current one than to a random vocabulary entry.

The parameter line breaks down as 112 for embeddings, which is 64 for the eight by eight token table plus 48 for six positions, and 1056 for the two blocks at 528 each. The output head is free. Exercise 3 asks you to do the same arithmetic at GPT-2 Small’s dimensions, and the answer comes to 85.0M of blocks plus 38.6M of token embeddings plus 0.8M of positional embeddings, which totals 124.3M against a published 124M, closing the accounting that Section 24.7 opened.

26.6 Autoregressive Generation

Training runs every position at once, which Section 26.3 described as the architecture’s main efficiency. Generation gets none of it, because the token at position 50 is an input to the computation at position 51 and does not exist until position 50 has produced it, so the parallelism that makes training cheap is simply unavailable at inference time. What replaces it is a loop. Feed the model whatever sequence exists so far, take the distribution at the final position and ignore all the others, choose a token from that distribution, append it, and go round again with a sequence one token longer than before.

/* 138_Generate.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#include <stdlib.h>

static float randf(void)
{
    return (float)rand() / RAND_MAX;
}
static float gelu(float x)
{
    float c = 0.7978846f;
    float u = x + 0.044715f * x * x * x;
    return 0.5f * x * (1.0f + tanhf(c * u));
}
static float dot(const float *a, const float *b, 
                 int n)
{
    float s = 0;
    int i;
    for (i = 0; i < n; i++) s += a[i] * b[i];
    return s;
}
static void softmax(float *x, int n)
{
    float mx = -FLT_MAX, s = 0;
    int i;
    for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
    for (i = 0; i < n; i++) {
        x[i] = expf(x[i] - mx);
        s += x[i];
    }
    for (i = 0; i < n; i++) x[i] /= s;
}
static void rmsnorm(float *x, int n)
{
    int i;
    double ss = 0;
    float eps = 1e-5f;
    for (i = 0; i < n; i++) ss += x[i] * x[i];
    float ri = 1.0f / sqrtf((float)(ss/n) + eps);
    for (i = 0; i < n; i++) x[i] *= ri;
}

#define VOCAB 8
#define MAX_SEQ 20
#define DM 4

/* Greedy, return the highest scoring token */
static int sample_greedy(const float *logits, int n)
{
    int best = 0, i;
    for (i = 1; i < n; i++)
        if (logits[i] > logits[best]) best = i;
    return best;
}

/* Temperature sampling: add randomness */
static int sample_temperature(float *logits, int n, 
                              float temp)
{
    int i;
    for (i = 0; i < n; i++) logits[i] /= temp;
    softmax(logits, n);

    /* Random sample from the distribution */
    float r = randf();
    float cumsum = 0;
    for (i = 0; i < n; i++) {
        cumsum += logits[i];
        if (r < cumsum) return i;
    }
    return n - 1;
}

int main(void)
{
    /* A tiny stand-in model, embedding plus
       output projection and nothing between */
    float embed[VOCAB][DM];
    int i, j;
    srand(42);
    for (i = 0; i < VOCAB; i++)
        for (j = 0; j < DM; j++)
            embed[i][j] = (randf()*2-1) * 0.5f;

    const char *names[] = { "<PAD>", "the", "cat",
                            "sat", "on", "a",
                            "mat", "<END>" };

    /* Prompt: "the cat" */
    int sequence[MAX_SEQ] = { 1, 2 };  /* the, cat */
    int cur_len = 2;
    int max_gen = 6;

    printf("Autoregressive generation:\n\n");
    printf("  Prompt: ");
    for (i = 0; i < cur_len; i++)
        printf("%s ", names[sequence[i]]);
    printf("\n\n");

    printf("  Generating (greedy):\n");
    for (int step = 0; step < max_gen; step++) {
        /* Use last token's embedding as the "model
           output"
           a real model would run every block here */
        int last = sequence[cur_len - 1];
        float logits[VOCAB];
        for (j = 0; j < VOCAB; j++)
            logits[j] = dot(embed[last], embed[j], DM);

        int next = sample_greedy(logits, VOCAB);
        sequence[cur_len++] = next;

        printf("    Step %d: predict '%s'", step,
               names[next]);
        if (next == 7) {
            printf(" (stop)\n");
            break;
            }
        printf("\n");
    }

    printf("\n  Full sequence: ");
    for (i = 0; i < cur_len; i++)
        printf("%s ", names[sequence[i]]);
    printf("\n");

    printf("\n  With temperature 0.8, same prompt\n");
    for (int trial = 0; trial < 3; trial++) {
        int seq2[MAX_SEQ] = { 1, 2 };
        int len2 = 2;
        printf("    Trial %d: the cat ", trial);
        for (int step = 0; step < 4; step++) {
            int last = seq2[len2-1];
            float logits[VOCAB];
            for (j = 0; j < VOCAB; j++)
                logits[j] = dot(embed[last], 
                    embed[j], DM);
            int next = sample_temperature(logits, 
                                          VOCAB, 0.8f);
            seq2[len2++] = next;
            printf("%s ", names[next]);
            if (next == 7) break;
        }
        printf("\n");
    }

    printf("\n  Temperature controls randomness:\n");
    printf("    temp=0 always takes the top token\n");
    printf("    temp=1 samples the raw distribution\n");
    printf("    temp>1 flattens it, more random\n");
    printf("    temp<1 sharpens it, "
           "more predictable\n");

    return 0;
}

The greedy run produces “the cat cat cat cat cat cat cat”, predicting the same token six times in a row, and the previous section already explained why. The prompt ends with “cat”, the untrained model copies its input, so the token after “cat” is “cat”, and then the token after that “cat” is “cat” again. Autoregressive generation feeds the model’s output back as its input, so a model that copies enters a fixed point immediately and never leaves.

Degenerate loops of exactly this kind occur in trained models too, which is why the failure is worth recognising here in its purest form. A trained model does not copy its input, but it can still find a phrase whose most likely continuation leads back to itself, and greedy decoding will then repeat that phrase until the token limit stops it. The cause is the same, since greedy decoding is deterministic and a deterministic map from state to state either terminates or cycles.

The temperature trials show the standard escape. Dividing the logits by a temperature before the softmax and then sampling rather than taking the maximum breaks the determinism, and the three trials here produce three different continuations from one prompt, running to “the cat”, “the cat sat the” and “the cat cat” before each hits the end token. Notice that the loop is gone in all three even though the model has not changed at all, since sampling only has to avoid the top token once to leave the fixed point. Temperature below 1 sharpens the distribution toward greedy behaviour and temperature above 1 flattens it toward uniform, and production systems typically sit between 0.7 and 1.0 with top-k or nucleus sampling layered on top, which exercises 1 and 2 ask you to implement.

The cost of this loop is the subject of the next chapter and it is worse than it first appears. Generating token 51 as written here means running the whole stack over all 51 positions, even though 50 of those positions computed exactly the same keys and values on the previous step, because the causal mask guarantees that nothing before position 50 can have been affected by anything after it. The same argument that Section 26.4 measured, that early positions are untouched by late ones, means the work done on them is identical every time and therefore entirely redundant. Worse, the waste grows with the length of the output, so the hundredth token costs about a hundred times the ninety ninth relative to what is actually new, and a long generation spends nearly all of its time recomputing a prefix it already knows. The KV cache in Chapter 27 stores those keys and values instead and turns the per token cost from quadratic in the length back to linear.

26.7 Encoder-Decoder Against Decoder-Only

Both architectures have now been built in full rather than described, so the comparison can be made on what they actually do rather than on reputation. Most of the differences are decisions rather than computations, and a decision is better set out in a table than wrapped in a program that prints it, so the summary below is a table and the one part that genuinely is arithmetic follows underneath it in prose.

The parameter arithmetic behind that first row is worth doing at realistic sizes. With d_model at 512 and d_ff at 2048, an encoder block costs 4 * d_model^2 for its single attention plus 2 * d_model * d_ff for its feedforward network, totalling 3.15M. A decoder block in the encoder-decoder arrangement carries two attention sublayers rather than one, so it costs 4.19M. One layer of the paired architecture is therefore 7.34M against 3.15M for one decoder-only layer, a ratio of 2.33 to 1.

Set that beside the training signal row and the case closes. The encoder-decoder arrangement spends 2.33 times the parameters per layer and extracts a gradient from a smaller fraction of the tokens it reads, in exchange for an architectural separation that language modelling has no use for. For translation the separation earns its cost, which is why encoder-decoder models remain in service there. For everything that can be phrased as continuing a text, which turned out to be almost everything, it does not.

26.8 Key Takeaways

• A decoder-only transformer drops the encoder and cross-attention, leaving a single stack of blocks with causal self-attention and a feedforward network. The input and the output are the same sequence.

• At position i the model predicts token i+1 from positions 0 through i. That convention turns any text into training data, with no need to label which part is input, which is what allowed training on uncurated text at scale.

• Every position yields a training signal, so a thousand token sequence gives 999 supervised examples in one forward pass. The sequence to sequence arrangement of Chapter 18 got a gradient only from the target side.

• The mask is what makes that parallelism safe. Section 26.4 verified it by changing only the last token of a sequence and measuring a change of exactly 0.00000 at every earlier position, against 1.42223 at the position that was edited.

• That test is worth keeping, because a broken mask produces no error and no warning. A model that can see its own answer learns to copy it and reports a very low training loss while generating gibberish.

• Weight tying reuses the embedding table transposed as the output head, so the same matrix maps tokens to vectors and vectors back to token scores. In GPT-2 Small that table is 38.6M against a 124M total.

• An untrained tied model copies its input. Section 26.5 predicted 3, 1, 4, 1, 5 from an input of 3, 1, 4, 1, 5, because near identity blocks leave the input embedding largely intact and a vector’s largest dot product is with itself.

• That copying explains the generation output too. Section 26.6 produced “cat” six times in a row, because greedy decoding feeds the output back as input and a model that copies reaches a fixed point on the first step.

• Trained models hit the same failure differently, finding a phrase whose most likely continuation returns to itself. Greedy decoding is deterministic, so it either terminates or cycles.

• Temperature sampling breaks the determinism by dividing the logits before the softmax and drawing rather than taking the maximum. Below 1 sharpens toward greedy, above 1 flattens toward uniform.

• One decoder-only layer costs 3.15M at d_model 512 and d_ff 2048, against 7.34M for an encoder and decoder layer pair, a ratio of 2.33 to 1.

• Generation as written re-runs the whole stack over every position for each new token, though nothing before the newest position can have changed. Chapter 27 removes that waste with the KV cache.

26.9 Exercises

1. Implement top-k sampling: instead of sampling from the full vocabulary, zero out all but the top-k logits, renormalize, and sample. Compare the output diversity to greedy and temperature sampling.

2. Implement top-p (nucleus) sampling: sort logits by probability, include tokens until the cumulative probability exceeds p, then sample from that subset.

3. Count the total parameters for a decoder-only model with d_model=768, d_ff=3072, 12 layers, vocab=50257, max_seq=1024. This is GPT-2 small. Compare to the published 124M.

4. Implement the training loop: for each position, compute cross-entropy between the predicted distribution and the actual next token. Sum across positions. Compute the gradient and update.

5. How much compute does generation require per token? Count the multiply-accumulate operations for one forward pass through one block. Multiply by the number of blocks.

6. In a trained model, what would the attention weights look like at the last position of the sequence “The cat sat on the ___”? Which positions would get the highest weight? Why?