Appendix B: Building and Running the Code
Every program in this book is a single self contained C file that depends on nothing but the standard library and libm. There is no build system, no package manager, and no configuration step. If you have a compiler you can run any listing in the book within a few seconds of reading it.
B.1 The Compile Line
| Toolchain | Command |
|---|---|
| GCC or Clang | cc -std=c99 -O2 -o prog prog.c -lm |
| MSVC | cl /O2 prog.c |
| Debug build | cc -std=c99 -g -O0 -o prog prog.c -lm |
The pieces matter more than the incantation. -std=c99 fixes the language version, and while most listings compile as C89 a few declare variables inside a for statement. -O2 turns on optimization, which for the training programs is the difference between two seconds and twenty. -lm links the math library, and leaving it off produces undefined references to expf, sqrtf and tanhf rather than a sensible error.
MSVC needs no math flag because its runtime includes those functions already. It will warn about scanf and friends under its own security rules, which you can ignore or silence with /D_CRT_SECURE_NO_WARNINGS.
B.2 Warnings Worth Turning On
| Flag | Catches |
|---|---|
| -Wall -Wextra | Unused variables, sign mismatches, missing initializers |
| -Wfloat-equal | Comparing floats with ==, which almost never means what you want |
| -Wshadow | An inner loop variable hiding an outer one, a real source of index bugs |
| -fsanitize=address | Reads and writes past the end of an array, at some runtime cost |
| -ffast-math | Nothing. Do not use it here |
The last row is a warning rather than a suggestion. Fast math permits the compiler to reorder floating point operations and to assume no value is ever NaN or infinite, which quietly changes results in exactly the numerical code this book is made of. The gradient checks in Chapter 37 will start failing and the cause will not be obvious.
Address sanitizer is worth keeping on while you are writing your own variations. Most of the arrays in these programs are fixed size and indexed by loop counters, so an off by one walks into a neighbouring tensor and produces wrong numbers rather than a crash. The sanitizer turns that into an immediate report.
B.3 Why the Programs Are Slow and When That Matters
Nothing in this book is optimized. A matrix multiply is three nested loops with no blocking, no vectorization by hand and no threading, because the point is that you can read it. At the sizes used in the chapters that costs nothing worth measuring, and the capstone still trains a full transformer in about ten seconds.
If you scale a program up and it becomes unbearable, the first thing to check is whether you left optimization off. The second is memory layout. Looping over the last index of a multidimensional array is fast and looping over the first is slow, sometimes by a factor of ten, because of how C lays arrays out in memory. Chapter 35 measures this directly.
B.4 Floating Point, and What to Expect
Every program uses float rather than double. That is the right choice, since it is what real inference uses and it halves the memory traffic, but it has consequences worth knowing about.
| Symptom | Cause |
|---|---|
| Your last digit differs from the book | Different compiler, different order of operations. Expected |
| A sum of many values drifts | Accumulate in double and store the result as float |
| A gradient check agrees only to 1e-3 | The limit of single precision differencing, not a bug |
| Loss becomes nan | Almost always a log or a divide reached zero. Add an epsilon |
| Loss becomes inf | An exponential overflowed. Subtract the maximum before expf |
The last two are the failures you will actually hit. A softmax written directly from the formula overflows the moment a logit exceeds about 88, which is why every softmax in this book subtracts the row maximum first. It changes nothing mathematically and it is the difference between working code and a model full of NaN.
B.5 Random Numbers
Programs that need random weights call rand seeded with a fixed value, so a given program produces the same result every time you run it on the same machine. It does not produce the same result on a different one. The C standard specifies the interface and not the algorithm, and the sequence from glibc differs completely from the sequence from the Microsoft runtime.
This is why figures in the book that come from programs using rand carry numbers you will not reproduce exactly. The text says so wherever it matters. The structure of the result, which is what the chapter is arguing about, does not change.
If you want figures that match across machines, replace rand with a small generator defined in the program itself. A 32 bit xorshift is four lines and gives an identical stream everywhere.
B.6 Running Every Program at Once
The listings are numbered sequentially across the whole book, from 001 through 185, and each carries its number in the comment on its first line. A shell loop is enough to build and run all of them.
for f in *.c; do
cc -std=c99 -O2 -o "${f%.c}" "$f" -lm || echo "FAILED $f"
done
for p in [0-9][0-9][0-9]_*; do
[ -x "$p" ] && { echo "== $p"; ./"$p"; }
done
Two programs take noticeably longer than the rest. The capstone in Chapter 37 trains a transformer and runs for roughly ten seconds, and the load balancing program in Chapter 36 runs four hundred routing passes. Everything else finishes faster than you can read its output.