Pretraining and Fine-Tuning
Two-phase training and transfer learning
31.1 What You Will Learn
Everything insofar trained a model once, on one dataset, for one purpose, and then stopped. That is not how any language model you have used was actually built. Real models go through two phases with entirely different data requirements, costs and failure modes, and understanding why that split exists explains a good deal of what the industry looks like from the outside. This also gives us important insight into why so few organizations train models and so many ship them.
The first phase reads an enormous quantity of ordinary text and learns nothing more specific than what usually comes next. There aren’t any labels, task definitions or humans anywhere writing down correct answers. This means only the next-token prediction that we establish later on turns any text at all into training data without a schema. This phase is ruinously expensive, consuming months of time on thousands of accelerators for a frontier model, and for a given set of weights it happens exactly once. The second phase takes those weights and continues training on a small, deliberate dataset aimed at a particular behavior. The model already knows the language, so what it needs is redirection rather than education, and redirection is relatively cheap. DeepSeek-V3 pretrained on 14.8 trillion tokens and then fine-tuned on 1.5 million instruction examples, a ratio of roughly ten million to one, and the compute split is nearly as lopsided.
This chapter builds both phases on a model small enough to watch so we can understand what’s going on inside. We’ll work on a pretrained corpus of a hundred characters and then fine tune it on a different pattern. In another section we’ll do the thing the first section carefully avoids, which is fine-tuning hard enough to destroy what pretraining achieved. That failure has a name, “catastrophic forgetting”, and the point of measuring it is that the dial controlling it is the same learning rate you were already choosing.
31.2 The Two Phases
Pretraining is self-supervised, which is a precise term rather than a euphemism. The labels exist, since every position has a correct next token, but nobody wrote them down, because they are already present in the text. That is what removes the ceiling on dataset size. Supervised learning needs someone to produce a label per example and human effort is finite, while next-token prediction turns a trillion tokens of scraped text into a trillion training examples at no marginal cost.
Fine-tuning is ordinary supervised learning on a small deliberate dataset, and the reason it works at all is that the pretrained weights are not a starting point in the usual sense. They already encode the structure of the language, so the optimizer is not searching for a working model, it is nudging a working model toward a particular behavior. That is why a few thousand examples can accomplish what would otherwise take millions.
The economics follow from the split. DeepSeek-V3′s pretraining consumed roughly 2,664,000 GPU hours and its fine-tuning around 5,000, a ratio of about 530 to 1, so the second phase is close to free against the first. That asymmetry is why a handful of organizations pretrain and a great many fine-tune, and why the same base model appears underneath dozens of differently behaved products.
| Pretraining | Fine-tuning | |
|---|---|---|
| Data | 14.8T tokens | 1.5M examples |
| Labels | none needed | written by hand |
| Compute | 2,664,000 GPU hours | about 5,000 |
| Frequency | once | repeatedly, per task |
| Learning rate | 2.2e-4 peak | 5e-6 down to 1e-6 |
The last row is the one this chapter ends up being about. A fine-tuning learning rate roughly forty times smaller than the pretraining peak is not a minor tuning detail, it is the mechanism that stops the second phase erasing the first, and later on we measure what happens without it.
31.3 Pretraining and Then Fine-Tuning
The model here is deliberately tiny. A character embedding, one hidden layer, and an output distribution over a small vocabulary, predicting the next character from the current one and nothing else. That single character of context makes it a bigram model in the classical sense, which is far too weak to write English and exactly strong enough to demonstrate the two phase structure end to end without any of the machinery of the previous chapters getting in the way of the point.
Pretraining runs on a hundred character corpus of sentences about cats and dogs. Fine-tuning then runs on a completely different pattern, the string “qu” repeated, teaching the model that ‘q’ is followed by ‘u’. The question the program asks at the end is whether learning the second thing cost it the first.
/* 155_Pretrain.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#include <float.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define VOCAB 28 /* a-z + space + period */
#define DM 16
#define N_HID 32
#define SEQ_LEN 8
static int char_to_id(char c)
{
if (c >= 'a' && c <= 'z') return c - 'a';
if (c == ' ') return 26;
if (c == '.') return 27;
return 26; /* unknown -> space */
}
static char id_to_char(int id)
{
if (id < 26) return 'a' + id;
if (id == 26) return ' ';
return '.';
}
/* Simple one-layer network: embed -> hidden ->
output */
typedef struct {
float embed[VOCAB][DM];
float W1[N_HID][DM];
float b1[N_HID];
float W2[VOCAB][N_HID];
float b2[VOCAB];
}
Model;
static void model_init(Model *m)
{
int i, j;
for (i = 0; i < VOCAB; i++)
for (j = 0; j < DM; j++)
m->embed[i][j] = (randf()*2-1) * 0.1f;
for (i = 0; i < N_HID; i++) {
m->b1[i] = 0;
for (j = 0; j < DM; j++)
m->W1[i][j] = (randf()*2-1) * 0.2f;
}
for (i = 0; i < VOCAB; i++) {
m->b2[i] = 0;
for (j = 0; j < N_HID; j++)
m->W2[i][j] = (randf()*2-1) * 0.2f;
}
}
static float relu(float x)
{
return x > 0 ? x : 0;
}
/* Forward: given previous chars, predict next char */
static void forward(const Model *m, int input_id,
float hidden[N_HID],
float logits[VOCAB])
{
int i, j;
for (i = 0; i < N_HID; i++) {
hidden[i] = m->b1[i];
for (j = 0; j < DM; j++)
hidden[i] +=
m->W1[i][j] * m->embed[input_id][j];
hidden[i] = relu(hidden[i]);
}
for (i = 0; i < VOCAB; i++) {
logits[i] = m->b2[i];
for (j = 0; j < N_HID; j++)
logits[i] += m->W2[i][j] * hidden[j];
}
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
/* Train one step: predict target from input,
backprop */
/* Average loss over a corpus. A single argmax is
too crude to tell whether pretraining survived,
so measure the whole distribution instead. */
static float corpus_loss(const Model *m, const char *d)
{
int len = (int)strlen(d), i;
float total = 0;
for (i = 0; i < len - 1; i++) {
float h[N_HID], lo[VOCAB];
forward(m, char_to_id(d[i]), h, lo);
softmax(lo, VOCAB);
total += -logf(lo[char_to_id(d[i+1])] + 1e-9f);
}
return total / (len - 1);
}
static float train_step(Model *m, int input_id,
int target_id, float lr)
{
float hidden[N_HID], logits[VOCAB], probs[VOCAB];
int i, j;
forward(m, input_id, hidden, logits);
memcpy(probs, logits, sizeof(logits));
softmax(probs, VOCAB);
float loss = -logf(probs[target_id] + 1e-8f);
/* d_loss/d_logits is probs - one_hot(target) */
float d_logits[VOCAB];
for (i = 0; i < VOCAB; i++)
d_logits[i] =
probs[i] - (i == target_id ? 1.0f : 0.0f);
/* Hidden gradients, taken from W2 as it stands
before any update touches it. Updating W2 first
and differentiating through the new values is a
bug that produces a wrong gradient silently. */
float d_hidden[N_HID];
for (j = 0; j < N_HID; j++) {
d_hidden[j] = 0;
for (i = 0; i < VOCAB; i++)
d_hidden[j] += d_logits[i] * m->W2[i][j];
if (hidden[j] <= 0) d_hidden[j] = 0; /* ReLU */
}
/* W2, b2 gradients */
for (i = 0; i < VOCAB; i++) {
m->b2[i] -= lr * d_logits[i];
for (j = 0; j < N_HID; j++)
m->W2[i][j] -= lr * d_logits[i] * hidden[j];
}
/* W1, b1, embedding gradients */
float d_embed[DM] = {0};
for (i = 0; i < N_HID; i++) {
m->b1[i] -= lr * d_hidden[i];
for (j = 0; j < DM; j++) {
d_embed[j] += d_hidden[i] * m->W1[i][j];
m->W1[i][j] -=
lr * d_hidden[i] * m
->embed[input_id][j];
}
}
for (j = 0; j < DM; j++)
m->embed[input_id][j] -= lr * d_embed[j];
return loss;
}
int main(void)
{
Model m;
srand(42);
model_init(&m);
/* Pretraining corpus: simple repeating patterns */
const char *corpus =
"the cat sat. the dog ran. "
"the cat ran. the dog sat. "
"a big cat. a big dog. "
"the big cat sat. a dog ran. ";
int corpus_len = strlen(corpus);
printf("=== PRETRAINING ===\n\n");
printf(" Corpus: \"%.*s...\"\n", 40, corpus);
printf(" Length: %d characters\n\n", corpus_len);
/* Train */
int epoch;
for (epoch = 0; epoch < 400; epoch++) {
float total_loss = 0;
int count = 0;
int i;
for (i = 0; i < corpus_len - 1; i++) {
int input = char_to_id(corpus[i]);
int target = char_to_id(corpus[i + 1]);
total_loss +=
train_step(&m, input, target, 0.01f);
count++;
}
if ((epoch + 1) % 100 == 0)
printf(" Epoch %3d: avg loss = %.3f\n",
epoch + 1, total_loss / count);
}
/* Test: generate from pretrained model */
printf("\n Generate from the pretrained model, "
"seed 't'\n ");
int cur = char_to_id('t');
printf("t");
for (int i = 0; i < 30; i++) {
float hidden[N_HID], logits[VOCAB];
forward(&m, cur, hidden, logits);
softmax(logits, VOCAB);
/* Greedy */
int best = 0;
for (int j = 1; j < VOCAB; j++)
if (logits[j] > logits[best]) best = j;
printf("%c", id_to_char(best));
cur = best;
}
printf("\n");
/* Save weights for fine-tuning */
printf("\n Pretrained weights saved. The model "
"has learned\n");
printf(" basic character patterns from the "
"corpus.\n");
/* === FINE-TUNING === */
float loss_before = corpus_loss(&m, corpus);
printf("\n=== FINE-TUNING ===\n\n");
/* New task, after 'q' always output 'u' */
const char
*ft_data = "qu qu qu qu qu qu qu qu qu qu ";
int ft_len = strlen(ft_data);
printf(" Fine-tuning data: \"%s\"\n", ft_data);
printf(" Task, learn that 'q' is always "
"followed by 'u'\n\n");
/* What does the model predict after 'q'? */
{
float hidden[N_HID], logits[VOCAB];
forward(&m, char_to_id('q'), hidden, logits);
softmax(logits, VOCAB);
int best = 0;
for (int j = 1; j < VOCAB; j++)
if (logits[j] > logits[best]) best = j;
printf(" Before fine-tuning: 'q' -> '%c' "
"at prob %.3f\n",
id_to_char(best), logits[best]);
printf(" P('u' | 'q') = %.3f\n",
logits[char_to_id('u')]);
}
/* Fine-tune with smaller learning rate */
for (epoch = 0; epoch < 50; epoch++) {
float total_loss = 0;
int count = 0;
for (int i = 0; i < ft_len - 1; i++) {
int input = char_to_id(ft_data[i]);
int target = char_to_id(ft_data[i + 1]);
total_loss +=
train_step(&m, input, target, 0.002f);
count++;
}
}
/* After fine-tuning */
{
float hidden[N_HID], logits[VOCAB];
forward(&m, char_to_id('q'), hidden, logits);
softmax(logits, VOCAB);
int best = 0;
for (int j = 1; j < VOCAB; j++)
if (logits[j] > logits[best]) best = j;
printf("\n After fine-tuning: 'q' -> '%c' "
"at prob %.3f\n",
id_to_char(best), logits[best]);
printf(" P('u' | 'q') = %.3f\n",
logits[char_to_id('u')]);
}
/* Did the pretraining survive? Report the loss
on the original corpus rather than guessing
from one prediction. */
printf("\n Loss on the pretraining corpus\n");
printf(" before fine-tuning: %.3f\n",
loss_before);
printf(" after fine-tuning: %.3f\n",
corpus_loss(&m, corpus));
printf("\n The new pattern was learned from 30\n");
printf(" characters of data. Whether the old\n");
printf(" corpus survived is the "
"loss above, and\n");
printf(" a rise there is the cost "
"of the change.\n");
return 0;
}
Figure 31-1 has both phases in it, and what the second one did to the first. Pretraining loss falls from 0.841 at a hundred epochs to 0.791 at four hundred, and the sample generated from a seed of ‘t’ loops on the word “dog”. That is not English and it is close to the best this architecture can do, because a bigram model sees only the current character. From ‘d’ it can only produce whatever most often follows ‘d’, which is ‘o’, then ‘g’ after ‘o’, then a space after ‘g’, and then nothing tells it what should come after a space except the most common option, so it loops. The loop is a property of one character of context, not a training failure.
The fine-tuning result on the new task is clean. Before, the model gives ‘q’ a most likely successor of ‘ ‘ at 0.325 and assigns ‘u’ a probability that rounds to zero. After a short pass over the “qu” data, ‘u’ comes out on top at 0.637. The new pattern was learned from thirty characters of data, which no model trained from scratch could manage on any budget, and it worked only because the weights being adjusted already encoded what a character was and how characters follow one another.
The final block is the one the chapter exists for, and it does not report what a first reading might expect. Loss on the original corpus rises from 0.755 before fine-tuning to 1.596 after, which is more than double. The model kept enough of the old corpus to be recognizable and it paid for the new pattern with a substantial part of what pretraining bought, and that happened at a fine-tuning rate of 0.002 against a pretraining rate of 0.01, already five times smaller.
Note what is being measured and why. An earlier version of this listing checked a single prediction, asking whether ‘t’ still produced ‘h’, and reported that it did. That check is far too crude, because one character can survive while the distribution behind it degrades badly, and it can also flip for reasons that have nothing to do with the fine-tuning. Loss over the whole corpus catches what an argmax misses, which is the same argument we make at greater length further down, with three learning rates instead of one.
31.4 Why Fine-Tuning Works
I want to state the reason plainly, because the phrase transfer learning tends to get used as though it explained itself. Pretraining does not produce a model that has memorized text. It produces a set of internal representations in which similar things are near each other and useful distinctions are already separated, and those representations are almost entirely task independent.
What pretraining supplies is the whole substrate. Which tokens co-occur, how grammar constrains what can follow what, whatever facts appeared often enough in the corpus to be worth encoding, patterns of reasoning that showed up across many documents, and the conventions of how text is formatted. None of that is specific to any downstream task and all of it has to exist before any downstream task is learnable. However, what fine-tuning supplies is narrow by comparison. Following an instruction rather than continuing a document, producing output in a particular shape such as JSON or code, refusing certain requests, and whatever domain vocabulary a specialist application needs. Each of those is a redirection of existing capability rather than a new capability, which is why the data requirement collapses from trillions to millions and the compute requirement from millions of GPU hours to thousands.
There is a corollary that matters for anyone choosing between the two. If a behavior depends on knowledge that was never in the pretraining corpus, fine-tuning will not install it, since there is nothing to redirect. Fine-tuning changes what a model does with what it knows, and it is a poor and expensive way to teach it something new.
31.5 Catastrophic Forgetting
The previous section fine-tuned gently and reported that the old knowledge survived. This section asks what happens when it does not, and it has to be careful about how the question is measured, because a single argmax prediction is far too crude to detect gradual damage. A model can lose most of what it knew while still getting one particular character right.
So the program measures the pretraining corpus twice over. Average cross-entropy loss is sensitive to the entire predicted distribution and will register damage long before any argmax flips, while accuracy is the blunt fraction of positions where the top prediction is correct. The program pretrains once, takes three copies of the resulting weights, fine-tunes each copy at a different learning rate, and reports for every copy both what it retained of the old corpus and what it managed to learn of the new task.
/* 156_Forgetting.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#include <float.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
static float relu(float x)
{
return x > 0 ? x : 0;
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
#define V 28
#define DM 16
#define NH 32
typedef struct {
float embed[V][DM];
float W1[NH][DM], b1[NH];
float W2[V][NH], b2[V];
}
Model;
static int c2i(char c)
{
if (c >= 'a' && c <= 'z') return c - 'a';
if (c == ' ') return 26;
return 27;
}
static char i2c(int i)
{
if (i < 26) return 'a' + i;
if (i == 26) return ' ';
return '.';
}
static void fwd(const Model *m, int id,
float h[NH], float lo[V])
{
int i, j;
for (i = 0; i < NH; i++) {
h[i] = m->b1[i];
for (j = 0; j < DM; j++)
h[i] += m->W1[i][j] * m->embed[id][j];
h[i] = relu(h[i]);
}
for (i = 0; i < V; i++) {
lo[i] = m->b2[i];
for (j = 0; j < NH; j++)
lo[i] += m->W2[i][j] * h[j];
}
}
static void train(Model *m, const char *data,
int epochs, float lr)
{
int len = strlen(data), ep, i, j, k;
for (ep = 0; ep < epochs; ep++)
for (i = 0; i < len - 1; i++) {
float h[NH], lo[V], pr[V];
int inp = c2i(data[i]), tgt = c2i(data[i+1]);
fwd(m, inp, h, lo);
memcpy(pr, lo, sizeof(lo));
softmax(pr, V);
float dl[V];
for (j = 0; j < V; j++)
dl[j] = pr[j] - (j == tgt ? 1.0f : 0.0f);
/* dh must come from W2 as it stands now, so it
is computed before W2 is touched. Updating
first and differentiating afterward is a bug
that quietly corrupts every later
gradient. */
float dh[NH];
for (j = 0; j < NH; j++) {
dh[j] = 0;
for (k = 0; k < V; k++)
dh[j] += dl[k] * m->W2[k][j];
if (h[j] <= 0) dh[j] = 0;
}
for (j = 0; j < V; j++) {
m->b2[j] -= lr * dl[j];
for (k = 0; k < NH; k++)
m->W2[j][k] -= lr * dl[j] * h[k];
}
float de[DM] = {0};
for (j = 0; j < NH; j++) {
m->b1[j] -= lr * dh[j];
for (k = 0; k < DM; k++) {
de[k] += dh[j] * m->W1[j][k];
m->W1[j][k] -=
lr * dh[j] * m->embed[inp][k];
}
}
for (j = 0; j < DM; j++)
m->embed[inp][j] -= lr * de[j];
}
}
static char predict(const Model *m, int inp)
{
float h[NH], lo[V];
fwd(m, inp, h, lo);
softmax(lo, V);
int best = 0;
for (int i = 1; i < V; i++)
if (lo[i] > lo[best]) best = i;
return i2c(best);
}
/* Average cross-entropy of the model on a corpus.
This is the number that measures forgetting, since
a single argmax says almost nothing. */
static float corpus_loss(const Model *m, const char *d)
{
int len = strlen(d), i, j;
float total = 0;
for (i = 0; i < len - 1; i++) {
float h[NH], lo[V];
int inp = c2i(d[i]), tgt = c2i(d[i+1]);
fwd(m, inp, h, lo);
softmax(lo, V);
total += -logf(lo[tgt] + 1e-9f);
}
return total / (len - 1);
}
/* Fraction of positions the model gets right */
static float corpus_acc(const Model *m, const char *d)
{
int len = strlen(d), i, j, hit = 0;
for (i = 0; i < len - 1; i++) {
float h[NH], lo[V];
int inp = c2i(d[i]),
tgt = c2i(d[i+1]), best = 0;
fwd(m, inp, h, lo);
for (j = 1; j < V; j++)
if (lo[j] > lo[best]) best = j;
if (best == tgt) hit++;
}
return 100.0f * hit / (len - 1);
}
static void init(Model *m)
{
int i, j;
srand(42);
for (i = 0; i < V; i++)
for (j = 0; j < DM; j++)
m->embed[i][j] = (randf()*2-1)*0.1f;
for (i = 0; i < NH; i++) {
m->b1[i] = 0;
for (j = 0; j < DM; j++)
m->W1[i][j] = (randf()*2-1)*0.2f;
}
for (i = 0; i < V; i++) {
m->b2[i] = 0;
for (j = 0; j < NH; j++)
m->W2[i][j] = (randf()*2-1)*0.2f;
}
}
int main(void)
{
Model base, m;
const char *pretrain_data =
"the cat sat. the dog ran. the cat ran. "
"the dog sat. a big cat. a big dog. "
"the big cat sat. a dog ran. ";
const char *ft_data =
"xyz xyz xyz xyz xyz xyz xyz xyz xyz xyz ";
printf("Catastrophic forgetting, measured\n\n");
init(&base);
printf(" before pretraining old loss %.3f "
"old acc %5.1f%%\n",
corpus_loss(&base, pretrain_data),
corpus_acc(&base, pretrain_data));
train(&base, pretrain_data, 300, 0.05f);
printf(" after pretraining old loss %.3f "
"old acc %5.1f%%\n\n",
corpus_loss(&base, pretrain_data),
corpus_acc(&base, pretrain_data));
printf(" Now fine-tune the same "
"pretrained model\n");
printf(" on 'xyz xyz ...' at three learning "
"rates.\n\n");
printf(" lr epochs old loss old acc "
"new acc\n");
printf(" ----- ------ -------- ------- "
"-------\n");
float rates[] = { 0.0005f, 0.005f, 0.05f };
int eps[] = { 50, 100, 200 };
int k;
for (k = 0; k < 3; k++) {
/* fresh copy each time */
m = base;
train(&m, ft_data, eps[k], rates[k]);
printf(" %.4f %6d %8.3f %6.1f%% "
"%6.1f%%\n",
rates[k], eps[k],
corpus_loss(&m, pretrain_data),
corpus_acc(&m, pretrain_data),
corpus_acc(&m, ft_data));
}
printf("\n Read the two accuracy "
"columns against\n");
printf(" each other. The gentlest "
"run keeps most\n");
printf(" of what pretraining taught and learns\n");
printf(" little. The harshest learns the new "
"task\n");
printf(" completely and throws "
"the old one away.\n");
printf(" That trade is the whole "
"of the problem,\n");
printf(" and the learning rate is the dial.\n");
return 0;
}
Figure 31-2 takes the same pretrained weights and fine-tunes them at three learning rates. Pretraining takes the corpus loss from 3.329 to 0.753 and accuracy from 7.9 percent to 66.3 percent. Both of those numbers deserve a moment. A loss of 3.329 at initialization is very close to the natural logarithm of the vocabulary size, which is 3.332 for 28 symbols and is exactly what a uniform guess produces, so the model genuinely started from nothing. And 66.3 percent is not a mediocre score, it is the exact ceiling for a bigram model on this corpus, computable by counting which character most often follows each other character. The model reached the best score its architecture allows.
The three fine-tuning runs then show the trade in a single table. At a learning rate of 0.0005 the model keeps all 66.3 percent on the old corpus, losing nothing measurable, and manages only 25.6 percent on the new task. At 0.005 it learns the new task perfectly at 100 percent and drops to 53.5 percent on the old one. At 0.05 it learns the same 100 percent, holds the same 53.5 percent, and sends the old corpus loss from 0.753 up to 4.079, which is worse than the 3.329 it started at before pretraining had happened at all.
That last figure is what catastrophic forgetting means in a sentence. The aggressively fine-tuned model is worse on the original corpus than an untrained one, which is active damage and not just forgetting, since the weights have been driven confidently toward a distribution that is wrong for that text. A random model is uncertain and a badly fine-tuned model is confidently mistaken, and the second is worse by every measure that matters.
The bottom two rows also make the case for measuring loss rather than accuracy. Both report 53.5 percent, so on accuracy alone the two runs look equally damaged and the extra order of magnitude in learning rate appears to have cost nothing. The loss column disagrees, rising from 2.811 to 4.079, which says the harsher run pushed the distribution considerably further from the corpus while happening to leave the same number of argmax predictions standing. Accuracy is a coarse instrument and it hides exactly this kind of degradation.
Notice also that the middle and bottom rows both reach 100 percent on the new task, so the extra learning rate bought no additional capability whatsoever and cost a large amount of loss. There is no reason to fine-tune harder than the point where the task is learned, and the usual practice of setting the fine-tuning rate ten to a hundred times below the pretraining rate is an approximation of finding that point without searching for it. DeepSeek-V3 pretrains at a peak of 2.2e-4 and fine-tunes from 5e-6, a factor of forty four.
31.6 Supervised Fine-Tuning for Instructions
The commonest fine-tuning task for a language model is instruction following, and the mechanism deserves describing precisely rather than in outline, because it contains one detail that summaries routinely leave out and that turns out to be the part doing the work.
The data is pairs and nothing more elaborate. An instruction such as “What is the capital of France?” paired with a response such as “The capital of France is Paris”, or an instruction to reverse a string paired with the reversed string. Training concatenates the two halves into a single sequence and runs the ordinary next-token objective across it, which is exactly the loss we built back in Chapter 26 and requires no new machinery of any kind. Everything that makes instruction tuning work is in the data rather than the algorithm.
The detail is the mask. Loss is computed only on the response tokens, not on the instruction tokens, even though both are present in the sequence and both are being predicted. Without the mask the model spends capacity learning to generate plausible instructions, which is not the behavior anybody wants, and worse, it dilutes the gradient that should be teaching it to answer. With the mask, gradient flows only through the part of the sequence the model will actually be asked to produce at inference time.
That is the whole of supervised fine-tuning, and its limits are the reason this book has two more chapters on training. A model that has been through SFT reliably follows the shape of an instruction and remains perfectly capable of following it with something harmful, confidently invented, or three paragraphs longer than anyone wanted. Nothing in the objective distinguishes a good response from a bad one, only a response from a non-response, since every example in the dataset is treated as equally correct.
Fixing that requires a signal saying which of two acceptable responses is the better one, and a next-token loss has no way to express such a thing, since it can only compare a prediction against a single correct answer. What is needed is a score rather than a target, which is precisely the situation reinforcement learning was built to handle when we covered it earlier, and two chapters from now we assemble the remaining pieces into a working alignment procedure.
31.7 Key Takeaways
Pretraining is self-supervised next-token prediction over unlabeled text, which removes the ceiling on dataset size because nobody has to write the labels down.
Fine-tuning is ordinary supervised learning on a small deliberate dataset, and it works because the pretrained weights are already a working model rather than a starting point.
DeepSeek-V3 pretrained on 14.8 trillion tokens and fine-tuned on 1.5 million examples, roughly ten million to one, with a compute ratio of about 530 to 1.
We pretrained a bigram model to a loss of 0.791 and then taught it ‘q’ followed by ‘u’ from thirty characters of data, raising P(u given q) from essentially zero to 0.637, while the loss on the original corpus rose from 0.755 to 1.596.
The generated sample loops because one character of context cannot do better. From ‘d’ the model can only produce whatever most often follows ‘d’, so it cycles rather than composing.
Measuring forgetting by a single argmax prediction is too crude to see gradual damage. We measured loss and accuracy across the whole pretraining corpus instead.
That model reached 66.3 percent accuracy after pretraining, which is the exact ceiling for a bigram model on that corpus, computable by hand from counting the commonest successor of each character.
Fine-tuning at 0.0005 kept all 66.3 percent and learned only 25.6 percent of the new task. At 0.005 it learned 100 percent of the new task and fell to 53.5 percent, and at 0.05 it learned the same 100 percent for the same 53.5 percent.
At the highest rate the corpus loss reached 4.079, worse than the 3.329 the model started with before pretraining. That is not forgetting but damage, since the weights are now confidently wrong rather than merely uncertain.
The top two accuracy figures were identical at 53.5 percent while their losses differed at 2.811 and 4.079, which is why forgetting has to be measured on the distribution rather than on how many argmax predictions happen to survive.
Fine-tuning harder than the point where the task is learned buys nothing and costs retention, which is what the usual rule of a rate ten to a hundred times below pretraining approximates.
Instruction tuning concatenates instruction and response and masks the loss to the response tokens only, so gradient flows through the part the model will actually be asked to generate.
SFT teaches the shape of a response and cannot distinguish a good one from a bad one, because every example in the dataset is treated as equally correct. Expressing a preference between two acceptable outputs needs a score rather than a target, which is where we are headed two chapters from now.
31.8 Exercises
Pretrain on a larger corpus (copy the training text 10x). Does the model generate more coherent text? How does loss compare?
Fine-tune on two different tasks sequentially. Does the model forget the first task after learning the second? Implement a simple multi-task fine-tuning that trains on both tasks alternately.
Compare fine-tuning the entire model vs freezing the embedding layer and only training W1, b1, W2, b2. Which preserves pretraining knowledge better?
Implement the loss mask for SFT, given a sequence with instruction tokens at positions 0-N and response tokens at positions N+1 to M, only compute cross-entropy loss at positions N+1 to M.
What is the minimum amount of fine-tuning data needed to learn the ‘q’ -> ‘u’ pattern? Try 1, 5, 10, 50 examples and measure accuracy.
DeepSeek-V3 uses “knowledge distillation from DeepSeek-R1” during fine-tuning. The R1 model generates reasoning chains that are used as training data for V3. Why is this better than training V3 on the raw task directly?