Appendix A: Derivative Reference

Every derivative in this table is derived somewhere in the chapters. They are collected here so you have one page open on a second monitor while writing a backward pass, instead of hunting through thirty of them. The chapter column tells you where the derivation lives when the result alone is not enough.

Notation throughout is the one the book uses. A gradient arriving from above is written $𝑔$ , the quantity a layer produced is $𝑦$ , and the quantity it received is $𝑥$ . Nothing here assumes a batch dimension, because none of the programs in this book carry one.

A.1 Activations

Name	Forward	Derivative	Ch.
Sigmoid	$𝜎 (𝑥) = \frac{1}{1 + 𝑒^{- 𝑥}}$	$𝜎 (𝑥) (1 - 𝜎 (𝑥))$	2
Tanh	$tanh (𝑥)$	$1 - {tanh}^{2} (𝑥)$	6
ReLU	$max (0, 𝑥)$	1 if $𝑥 > 0$ , else 0	5
Leaky ReLU	$max (𝛼 𝑥, 𝑥)$	1 if $𝑥 > 0$ , else $𝛼$	5
SiLU	$𝑥 𝜎 (𝑥)$	$𝜎 (𝑥) (1 + 𝑥 (1 - 𝜎 (𝑥)))$	24
GELU	$𝑥 Φ (𝑥)$	see A.2	24
Softmax	$\frac{𝑒^{𝑧_{𝑖}}}{\sum_{𝑘} 𝑒^{𝑧_{𝑘}}}$	see A.3	4

The rectifier has no derivative at zero. Every implementation picks one, and the book picks zero, which matters less than being consistent about it.

A.2 GELU

The book uses the tanh approximation, which is what GPT-2 uses as well as some models after it. Write $𝑢 = 𝑐 (𝑥 + 𝑎 𝑥^{3})$ with $𝑐 = \sqrt{2 / 𝜋} \approx 0.7978846$ and $𝑎 = 0.044715$ .

GELU (𝑥) = \frac{1}{2} 𝑥 (1 + tanh 𝑢)

{GELU}^{'} (𝑥) = \frac{1}{2} (1 + tanh 𝑢) + \frac{1}{2} 𝑥 (1 - {tanh}^{2} 𝑢) 𝑐 (1 + 3 𝑎 𝑥^{2})

The second term is the one that gets dropped. Dropping it leaves a derivative that is correct at large positive $𝑥$ , wrong near the origin, and plausible enough that training still appears to work.

A.3 Softmax and Cross Entropy

Taken separately, the softmax Jacobian is a full matrix and the cross entropy derivative divides by a probability that can be very small. Taken together, almost everything cancels.

\frac{𝜕 𝑝_{𝑖}}{𝜕 𝑧_{𝑗}} = 𝑝_{𝑖} (𝛿_{𝑖 𝑗} - 𝑝_{𝑗})

\frac{𝜕 𝐿}{𝜕 𝑧_{𝑗}} = 𝑝_{𝑗} - 𝑦_{𝑗}

The second line is the one to remember and the one the code uses. The predicted distribution minus the one hot target, with no division and nothing to guard against. When the loss is a mean over positions rather than a sum, the gradient carries the same divisor, and forgetting that scales every gradient in the model by the sequence length at once.

The first line is still needed wherever softmax appears without cross entropy behind it, which in this book means inside attention. There it is applied as a vector operation rather than a matrix.

\nabla 𝑧_{𝑗} = 𝑝_{𝑗} (\nabla 𝑝_{𝑗} - \sum_{𝑘} 𝑝_{𝑘} \nabla 𝑝_{𝑘})

A.4 Normalization

Both normalizers divide by a statistic computed from every component of the input, which is why the backward pass has a correction term rather than being elementwise. Write $\hat{𝑥}$ for the normalized activation and $𝑔$ for the learned gain.

Name	Forward	Ch.
LayerNorm	$𝑔 \cdot \frac{𝑥 - 𝜇}{\sqrt{𝜎^{2} + 𝜀}}$	23
RMSNorm	$𝑔 \cdot \frac{𝑥}{\sqrt{mean (𝑥^{2}) + 𝜀}}$	23

RMSNorm drops the mean subtraction, which removes one pass over the data and, as it turns out, costs nothing in quality. Every model in the second half of this book uses it.

\nabla 𝑔_{𝑖} = \nabla 𝑦_{𝑖} {\hat{𝑥}}_{𝑖}

\nabla 𝑥_{𝑗} = \frac{1}{𝜎} (𝑔_{𝑗} \nabla 𝑦_{𝑗} - {\hat{𝑥}}_{𝑗} \cdot \frac{1}{𝑛} \sum_{𝑖} 𝑔_{𝑖} \nabla 𝑦_{𝑖} {\hat{𝑥}}_{𝑖})

The sum inside the second expression is what couples the components. Leave it out and the gradient points in roughly the right direction, which is the worst possible failure mode because the loss still falls.

A.5 Linear Layers and Attention

For $𝑦 = 𝑊 𝑥$ with $𝑊$ of shape $𝑚 \times 𝑛$ , the three gradients are the ones every backward pass is assembled from.

\nabla 𝑊_{𝑖 𝑗} = \nabla 𝑦_{𝑖} 𝑥_{𝑗}, \nabla 𝑥_{𝑗} = \sum_{𝑖} 𝑊_{𝑖 𝑗} \nabla 𝑦_{𝑖}

Attention is these three applied five times with a softmax in the middle. Scores are $𝑠_{𝑖 𝑗} = 𝑞_{𝑖} \cdot 𝑘_{𝑗} / \sqrt{𝑑_{ℎ}}$ , weights are $𝑝_{𝑖} = softmax (𝑠_{𝑖})$ over the unmasked positions, and the context is $𝑐_{𝑖} = \sum_{𝑗} 𝑝_{𝑖 𝑗} 𝑣_{𝑗}$ .

Quantity	Gradient
$\nabla 𝑣_{𝑗}$	$\sum_{𝑖} 𝑝_{𝑖 𝑗} \nabla 𝑐_{𝑖}$
$\nabla 𝑝_{𝑖 𝑗}$	$\nabla 𝑐_{𝑖} \cdot 𝑣_{𝑗}$
$\nabla 𝑠_{𝑖 𝑗}$	softmax backward applied to $\nabla 𝑝_{𝑖}$ , then divided by $\sqrt{𝑑_{ℎ}}$
$\nabla 𝑞_{𝑖}$	$\sum_{𝑗} \nabla 𝑠_{𝑖 𝑗} 𝑘_{𝑗}$
$\nabla 𝑘_{𝑗}$	$\sum_{𝑖} \nabla 𝑠_{𝑖 𝑗} 𝑞_{𝑖}$

A masked position must receive no gradient. The cleanest way to guarantee that is to loop only as far as the diagonal in the backward pass exactly as the forward pass did, rather than computing everything and zeroing afterwards.

A.6 Residuals and Embeddings

A residual connection passes its gradient through unchanged, so the sublayer gradient adds to what is already flowing rather than replacing it. That single property is why deep stacks train, and it is one line of code.

An embedding lookup is a matrix multiply by a one hot vector, so its gradient is a scatter add. The row for the token that appeared accumulates the incoming gradient, and every other row receives nothing. A token appearing several times in one window accumulates several contributions, which is correct and is easy to get wrong by assigning rather than adding.

When the output head is tied to the embedding table, that table receives gradient from two directions in the same step, once as a readout and once as an input. Both contributions must be present. With only one of them the two roles pull against each other and the model cannot train, which Chapter 37 measures directly.

A.7 Optimizers

Name	Update	Ch.
SGD	$𝑤 \leftarrow 𝑤 - 𝜂 \nabla 𝑤$	5
Momentum	$𝑣 \leftarrow 𝛽 𝑣 + \nabla 𝑤$	8
Adam	see below	8

Adam keeps a running mean and a running uncentered variance of the gradient, corrects both for the bias introduced by starting them at zero, and divides one by the square root of the other.

𝑚 \leftarrow 𝛽_{1} 𝑚 + (1 - 𝛽_{1}) \nabla 𝑤, 𝑣 \leftarrow 𝛽_{2} 𝑣 + (1 - 𝛽_{2}) {(\nabla 𝑤)}^{2}

𝑤 \leftarrow 𝑤 - 𝜂 \frac{𝑚 / (1 - 𝛽_{1}^{𝑡})}{\sqrt{𝑣 / (1 - 𝛽_{2}^{𝑡})} + 𝜀}

Defaults of $𝛽_{1} = 0.9$ , $𝛽_{2} = 0.999$ and $𝜀 = 10^{- 8}$ are what the paper proposed and what almost everyone still uses. The capstone uses them unchanged.

A.8 Verifying Any of This

None of the formulas above are worth trusting in your own code until you have checked them against the loss they claim to differentiate. Perturb a weight by a small amount in both directions, measure how the loss responds, and compare.

\nabla 𝑤 \approx \frac{𝐿 (𝑤 + 𝜀) - 𝐿 (𝑤 - 𝜀)}{2 𝜀}

Two practical notes. Single precision arithmetic will limit the agreement to a few parts in a thousand, because the difference of two nearly equal float values throws away most of the significant digits. And a per element check is noisiest exactly where the gradients are smallest, which is usually the query and key projections. Perturbing a whole parameter block along one random direction and comparing against the analytic gradient projected onto that direction avoids both problems, and Chapter 37 shows the code.