RLHF and Alignment
PPO, GRPO, and training models to follow human preferences
33.1 What You Will Learn
Our foray into pretraining and fine-tuning in Chapter 31 though much needed, ended on a limitation rather than an achievement. A model that has been through supervised fine-tuning follows the shape of an instruction reliably, and nothing in the objective it was trained on can tell a good answer from a bad one. Every example in an SFT dataset is treated as equally correct, so the model learns what a response looks like and never learns which of two plausible responses a person would rather have received. Expressing that preference needs a score rather than a target, which is the situation we built when we explored reinforcement learning. The difficulty is that nobody can write down a reward function for helpfulness. What people can do reliably is look at two responses and say which is better, which is the premise behind Reinforcement Learning from Human Feedback or RLHF, and the whole of RLHF is machinery for turning that comparison into a gradient.
The pipeline however has three stages. Collect comparisons from people, fit a model that predicts which response a person would pick, then optimize the language model against that predicted score while keeping it tethered to where it started. The tether matters more than it sounds, since a policy optimizing an imperfect reward will find the places where the reward is wrong.
This chapter builds the reward model first, then Proximal Policy Optimization or PPO, a technique popularized by OpenAI. PPO is the algorithm that made RLHF work in practice. However as we shall see it carries a cost that becomes prohibitive at scale. The Group Relative Policy Optimization or GRPO, is DeepSeek-V3′s answer to that cost. They sit in that order because each one exists to solve a problem created by the one before it, so reading them out of sequence loses the argument. With that being said, let’s get into it.
33.2 The Pipeline
Look at Figure 33-1 as it lays the pipeline out. The three boxes across the middle are the stages, what each one consumes is underneath, and the dashed box above stage three is the piece that is easy to leave off a diagram and impossible to leave out of an implementation.
Stage one is data collection and it is the expensive part, since it is the only stage that requires our (human) time. A prompt goes to the model, several responses come back, and a person reads them and says which they prefer. What comes out of that process is a set of pairs, each one recording that a particular response was preferred to another for the same prompt, and carrying nothing else. There isn’t a numeric score attached, no explanation of the reasoning, and no indication of how much better the winner was. An ordering is all a person can supply reliably, and it is all the rest of the pipeline gets to work from.
Stage two turns those orderings into a function. A reward model takes a prompt and a response and returns a single number, trained so that preferred responses score higher than rejected ones. This is where the pairwise data becomes something an optimizer can use, and in the next section we look at the one property of that translation that surprises people.
Stage three optimizes the language model against the reward model, and it is reinforcement learning in the sense we looked at in the earlier chapter. The policy is the model, an action is a token, and the reward arrives at the end of a generation rather than at each step. The constraint that makes it work is a penalty for drifting away from the SFT model, which exists because the reward model is a learned approximation and a sufficiently determined optimizer will exploit the places where the approximation fails.
33.3 The Reward Model
What you should know is that the reward model here is deliberately transparent so that its learned parameters can be read directly. A response is described by four hand-specified features which are: helpfulness, accuracy, conciseness and safety. Essentially the model is nothing more than a weighted sum of those four numbers plus a bias term. A real reward model is a transformer that reads raw text and emits a score from a final linear layer, which is a great deal more capable and trained by exactly the same objective against exactly the same kind of data.
That objective is what is known as the Bradley-Terry model, which converts a difference of scores into a probability that one response beats the other. The probability that A is preferred to B is the sigmoid of score A minus score B. The loss is the negative log of that probability for the ordering a person actually chose. Note what the loss depends on, since the whole of this section turns on it.
/* 161_Reward_Model.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float sigmoid(float z)
{
return 1.0f / (1.0f + expf(-z));
}
#define FEAT_DIM 4
/* Simple reward model: linear function of features */
typedef struct {
float w[FEAT_DIM];
float bias;
}
RewardModel;
static float reward_score(const RewardModel *rm,
const float f[FEAT_DIM])
{
float score = rm->bias;
int i;
for (i = 0; i < FEAT_DIM; i++)
score += rm->w[i] * f[i];
return score;
}
/* Train on preference pairs using the Bradley-Terry
model:
P(response_a > response_b) =
sigmoid(score_a - score_b)
Loss =
-log(sigmoid(score_chosen - score_rejected)) */
static float train_step(RewardModel *rm,
const float chosen[FEAT_DIM],
const float rejected[FEAT_DIM],
float lr)
{
float s_c = reward_score(rm, chosen);
float s_r = reward_score(rm, rejected);
float diff = s_c - s_r;
float prob = sigmoid(diff);
float loss = -logf(prob + 1e-8f);
/* d_loss/d_diff = -(1 - prob) = prob - 1 */
float grad = prob - 1.0f;
int i;
for (i = 0; i < FEAT_DIM; i++) {
rm->w[i] -= lr * grad
* (chosen[i] - rejected[i]);
}
/* The bias gets no update, and that is not an
oversight. It appears in both scores, so it
cancels out of the difference the loss is
built from, and its gradient is exactly
1 - 1 = 0. Pairwise data cannot pin down an
absolute offset, only relative ones. */
return loss;
}
int main(void)
{
RewardModel rm = { .w = {0}, .bias = 0 };
/* Preference data: features represent response
quality
[helpfulness, accuracy, conciseness, safety] */
float pairs[][2][FEAT_DIM] = {
/* chosen, rejected */
{{ 0.9f, 0.8f, 0.7f, 1.0f },
{ 0.3f, 0.2f, 0.5f, 0.8f }},
{{ 0.7f, 0.9f, 0.6f, 1.0f },
{ 0.8f, 0.1f, 0.9f, 0.3f }},
{{ 0.8f, 0.7f, 0.8f, 0.9f },
{ 0.2f, 0.6f, 0.3f, 1.0f }},
{{ 0.6f, 0.8f, 0.5f, 1.0f },
{ 0.9f, 0.7f, 0.8f, 0.1f }},
{{ 0.8f, 0.9f, 0.7f, 0.9f },
{ 0.4f, 0.3f, 0.6f, 0.7f }},
};
int n_pairs = 5;
printf("Training a reward model on preference "
"pairs\n\n");
int epoch, p;
for (epoch = 0; epoch < 200; epoch++) {
float total_loss = 0;
for (p = 0; p < n_pairs; p++)
total_loss += train_step(&rm, pairs[p][0],
pairs[p][1], 0.1f);
if ((epoch + 1) % 50 == 0) {
/* Check accuracy */
int correct = 0;
for (p = 0; p < n_pairs; p++) {
float sc = reward_score(&rm,
pairs[p][0]);
float sr = reward_score(&rm,
pairs[p][1]);
if (sc > sr) correct++;
}
printf(" Epoch %3d: loss=%.3f "
"accuracy=%d/%d\n",
epoch + 1, total_loss / n_pairs,
correct, n_pairs);
}
}
printf("\nLearned weights:\n");
const char *names[] = { "helpfulness", "accuracy",
"conciseness", "safety" };
for (int i = 0; i < FEAT_DIM; i++)
printf(" %-14s %+.3f\n", names[i], rm.w[i]);
printf(" bias %+.3f <- cannot move\n",
rm.bias);
printf("\nThe bias stays where it started. It "
"sits\n");
printf("in both scores and cancels in the\n");
printf("difference, so preference data cannot\n");
printf("say anything about it at all.\n\n");
printf("The weights are what humans valued.\n");
printf("Safety scores high because humans kept\n");
printf("preferring the safe response.\n");
return 0;
}
Figure 33-2 shows the reward model learning from five preference pairs. Training converges quickly, with the loss falling from 0.142 to 0.044 over two hundred epochs and every one of the five pairs ordered correctly from epoch 50 onward. The learned weights read 2.760 for helpfulness, 3.339 for accuracy, 1.144 for conciseness and 3.755 for safety, so safety and accuracy dominate, which is what the training pairs were built to reward.
The bias reads exactly zero and the listing says it cannot move. That is the property worth understanding, and the original version of this program got it wrong in a way that produced a striking wrong answer. The loss depends only on the difference between two scores, and the bias appears in both, so it cancels completely. Its derivative is one minus one, which is zero, and no amount of training data will ever shift it. The consequence is that a reward model trained on comparisons has no absolute scale. It can tell you that this response is better than that one and it cannot tell you whether either is any good, because adding a hundred to every score doesn’t change the prediction it makes. That is not a defect in the implementation, it is what preference data contains, and it means the numbers a reward model emits are meaningful only relative to other numbers from the same model.
A later section in this same chapter leans on exactly this. GRPO normalizes rewards within a group by subtracting the group mean, which throws away the absolute level entirely, and it can do that without losing anything because the absolute level was never information to begin with. An operation that looks like it discards signal turns out to discard only an arbitrary offset. We’ll get to that later on but for now let’s look at PPO.
33.4 PPO
Proximal Policy Optimization is the algorithm that made RLHF work at scale, and its central device is a limit on how far a single update is allowed to move the policy. It computes the ratio between the probability the new policy assigns to an action and the probability the old policy assigned to that same action, multiplies the ratio by the advantage, and then computes the same product again with the ratio clipped into a narrow band around 1. Whichever of the two is smaller becomes the objective, which means an update is only ever trusted up to the point where the clip takes over.
The reason for the ratio is worth being concrete about, because it explains the whole structure. Generating text is expensive, so PPO reuses a single batch of generations for several gradient steps rather than one. After the first of those steps the policy has moved, so the data was collected under a policy that no longer exists, and the ratio corrects for that mismatch. The clip stops the correction being trusted too far when the two policies have drifted apart. Take a look at this program.
/* 162_Ppo.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
/* Simplified PPO for a discrete action space.
The key idea: clipped surrogate objective.
ratio = pi_new(a|s) / pi_old(a|s)
advantage = reward - baseline
L = min(ratio * adv,
clip(ratio, 1-eps, 1+eps) * adv)
The clipping prevents the policy from changing
too much
in a single update. */
static float clip(float x, float lo, float hi)
{
if (x < lo) return lo;
if (x > hi) return hi;
return x;
}
#define N_ACTIONS 4
#define N_INNER 4
#define N_STEPS 10
#define N_SAMPLES 8
static void softmax(float *x, int n)
{
float mx = -1e9f, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
int main(void)
{
/* Policy logits (what we are optimizing) */
float logits[N_ACTIONS] = { 0.5f, 0.3f,
-0.2f, 0.1f };
/* Old policy (frozen for ratio computation) */
float old_logits[N_ACTIONS];
/* Rewards for each action (from the reward
model) */
float rewards[N_ACTIONS] = { 2.0f, 5.0f,
1.0f, 3.0f };
/* Action 1 has the highest reward */
float eps = 0.2f; /* clipping range */
int clips = 0;
float max_ratio = 0;
float lr = 0.1f;
int step, i;
srand(42);
printf("PPO training (simplified):\n\n");
printf(" Rewards: [%.1f, %.1f, %.1f, %.1f], "
"action 1 is best\n\n",
rewards[0], rewards[1],
rewards[2], rewards[3]);
for (step = 0; step < N_STEPS; step++) {
/* Save old policy */
for (i = 0; i < N_ACTIONS; i++)
old_logits[i] = logits[i];
float old_probs[N_ACTIONS],
new_probs[N_ACTIONS];
for (i = 0; i < N_ACTIONS; i++)
old_probs[i] = old_logits[i];
softmax(old_probs, N_ACTIONS);
/* Baseline, the reward the old policy
expects to collect */
float baseline = 0;
for (i = 0; i < N_ACTIONS; i++)
baseline += old_probs[i] * rewards[i];
/* PPO reuses one batch of data for several
gradient steps. That reuse is the whole
reason a ratio exists, since after the
first inner step the policy has moved and
the ratio is no longer 1. */
int inner;
for (inner = 0; inner < N_INNER; inner++) {
for (i = 0; i < N_ACTIONS; i++)
new_probs[i] = logits[i];
softmax(new_probs, N_ACTIONS);
for (i = 0; i < N_ACTIONS; i++) {
float ratio =
new_probs[i]
/ (old_probs[i] + 1e-8f);
float adv = rewards[i] - baseline;
float cl =
clip(ratio, 1.0f - eps, 1.0f + eps);
float surrogate =
fminf(ratio * adv, cl * adv);
if (ratio < 1.0f - eps
|| ratio > 1.0f + eps) clips++;
if (ratio > max_ratio)
max_ratio = ratio;
logits[i] +=
lr * surrogate
* (1.0f - new_probs[i]);
}
}
if ((step + 1) % 2 == 0) {
for (i = 0; i < N_ACTIONS; i++)
new_probs[i] = logits[i];
softmax(new_probs, N_ACTIONS);
printf(" Step %2d: probs=[%.3f, %.3f, "
"%.3f, "
"%.3f] max ratio %.2f\n",
step + 1, new_probs[0],
new_probs[1],
new_probs[2],
new_probs[3], max_ratio);
max_ratio = 0;
}
}
printf("\n The policy moves toward action 1, "
"which\n");
printf(" pays the most.\n\n");
printf(" The clip fired %d times out of %d "
"checks.\n",
clips, N_STEPS * N_INNER * N_ACTIONS);
printf(" Each outer step reuses "
"one batch for %d\n",
N_INNER);
printf(" gradient steps, and the ratio drifts "
"away\n");
printf(" from 1 as it does. With a single step "
"per\n");
printf(" batch the ratio would be exactly 1 "
"every\n");
printf(" time and the clip would never bind.\n");
return 0;
}
Figure 33-3 tracks the ratio as it drifts and the clip as it fires. As we can see from the results, the probability of the best action climbs from 0.627 at step 2 to 0.997 at step 10, which is the policy doing what a policy should. The column that matters more is the maximum ratio, reading 1.52 at step 2 and falling through 1.17, 1.06 and 1.02 to 1.00 at the end, and the clip fired 68 times out of 160 checks.
Those two numbers are the section’s point and the original listing could not have produced either. As written, it snapshotted the old policy and then computed the new probabilities from the same unchanged parameters. What this means is that the ratio was exactly 1.0 at every single check and the clip never once bound. The program asserted that clipping prevented the policy jumping too fast while the clipping code was unreachable in practice. What was missing is the inner loop. Four gradient steps per batch of data is what makes the ratio drift, and the drift is largest early when the policy is furthest from optimal and has the most to gain from a large move, which is exactly when an unclipped update would overshoot. The ratio falling to 1.00 by step 10 says the policy has nearly converged, so successive inner steps barely move it and the clip stops mattering.
The last thing I want to cover in this section before we move on is the cost. PPO needs an advantage, and computing an advantage needs a baseline estimate of how good a state is, which in the standard formulation is a critic network usually the same size as the policy. For a 671 billion parameter model that means holding and training two 671 billion parameter models, and the memory does not exist. That is the problem the next section solves by deleting the critic so we’ll discuss that now.
33.5 GRPO
The guys at DeepSeek came up with a very clever solution because DeepSeek-V3 removes the critic entirely, and the observation that permits it is simple enough to state in one sentence. A baseline is only something to compare a reward against, and if you have generated several responses to the same prompt then you are already holding several rewards for that prompt. So that means the group can supply its own comparison and a learned value function is not needed. The generations were required anyway, so the baseline arrives essentially free. Take a look at this equation:
Each response’s advantage is how far above or below the group average it scored, measured in units of the group’s own spread. A response better than its siblings gets a positive advantage and is made more likely, one worse gets a negative advantage and is made less likely, and there is no learned value function anywhere in the calculation. Everything on the right hand side is arithmetic over numbers already in hand. enough theory though let’s see the GRPO in practice.
/* 163_Grpo.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define GROUP_SIZE 6
#define N_ACTIONS 4
static void softmax(float *x, int n)
{
float mx = -1e9f, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
/* Sample an action from probability distribution */
static int sample(const float *probs, int n)
{
float r = randf(), cum = 0;
int i;
for (i = 0; i < n; i++) {
cum += probs[i];
if (r < cum) return i;
}
return n - 1;
}
/* Simulate a reward model: score depends on action */
static float get_reward(int action)
{
float base_rewards[] = { 2.0f, 5.0f, 1.0f, 3.0f };
/* Add noise to simulate reward model uncertainty */
return base_rewards[action]
+ (randf() - 0.5f) * 1.0f;
}
int main(void)
{
/* uniform start */
float logits[N_ACTIONS] = { 0, 0, 0, 0 };
/* the reference policy, which is the SFT model */
float ref_logits[N_ACTIONS] = { 0, 0, 0, 0 };
float epsilon = 0.2f;
float beta = 0.1f; /* KL penalty weight */
float lr = 0.3f;
int step, i, g;
srand(42);
printf("GRPO training (DeepSeek-V3 style):\n\n");
printf(" Group size: %d responses per prompt\n",
GROUP_SIZE);
printf(" Rewards: action 1 is best (~5.0)\n\n");
for (step = 0; step < 20; step++) {
float probs[N_ACTIONS];
for (i = 0; i < N_ACTIONS; i++)
probs[i] = logits[i];
softmax(probs, N_ACTIONS);
/* Generate a group of responses */
int actions[GROUP_SIZE];
float rewards[GROUP_SIZE];
for (g = 0; g < GROUP_SIZE; g++) {
actions[g] = sample(probs, N_ACTIONS);
rewards[g] = get_reward(actions[g]);
}
/* Compute group statistics */
float mean_r = 0, std_r = 0;
for (g = 0; g < GROUP_SIZE; g++)
mean_r += rewards[g];
mean_r /= GROUP_SIZE;
for (g = 0; g < GROUP_SIZE; g++) {
float d = rewards[g] - mean_r;
std_r += d * d;
}
std_r = sqrtf(std_r / GROUP_SIZE + 1e-8f);
/* GRPO update: advantage is group-relative */
for (g = 0; g < GROUP_SIZE; g++) {
float advantage =
(rewards[g] - mean_r) / std_r;
int a = actions[g];
/* Policy gradient with clipping */
float ref_probs[N_ACTIONS];
for (i = 0; i < N_ACTIONS; i++)
ref_probs[i] = ref_logits[i];
softmax(ref_probs, N_ACTIONS);
/* Update logit for the chosen action */
float ratio = probs[a]
/ (ref_probs[a] + 1e-8f);
float clipped = ratio;
if (clipped > 1.0f + epsilon)
clipped = 1.0f + epsilon;
if (clipped < 1.0f - epsilon)
clipped = 1.0f - epsilon;
float obj = fminf(ratio * advantage,
clipped * advantage);
/* KL penalty: keep close to reference */
float kl = ref_probs[a] / (probs[a] + 1e-8f)
- logf(ref_probs[a]
/ (probs[a] + 1e-8f)) - 1.0f;
logits[a] +=
lr * (obj - beta * kl) / GROUP_SIZE;
}
if ((step + 1) % 5 == 0) {
/* Both numbers must describe the same
policy, so exp is computed from the
probabilities the group was drawn
from rather than from the updated
ones. Otherwise the columns lag by a
step and the gap is not just noise. */
float base[] = { 2.0f, 5.0f, 1.0f, 3.0f };
float expected = 0;
for (i = 0; i < N_ACTIONS; i++)
expected += probs[i] * base[i];
printf(" Step %2d: probs=[%.2f %.2f %.2f "
"%.2f] exp %.2f group %.2f\n",
step + 1, probs[0], probs[1],
probs[2], probs[3],
expected, mean_r);
}
}
printf("\n Both columns describe the same "
"policy.\n");
printf(" exp is what it earns "
"on average, worked\n");
printf(" out exactly. group is an estimate of\n");
printf(" that same quantity from %d draws, and\n",
GROUP_SIZE);
printf(" the two differ by pure "
"sampling noise.\n");
printf(" That noise is what the "
"baseline carries\n");
printf(" into every advantage, "
"and it is why the\n");
printf(" group size is a real choice.\n\n");
printf(" GRPO advantages over PPO:\n");
printf(" - No critic, saving a whole model\n");
printf(" - Group statistics as the baseline\n");
printf(" - Takes mixed reward "
"sources easily\n\n");
printf(" DeepSeek-V3 uses GRPO with:\n");
printf(" - Rule rewards for math and code\n");
printf(" - Model rewards for open-ended work\n");
printf(" - Self-rewarding by voting\n");
return 0;
}
Figure 33-4 shows GRPO using the group as its own baseline. The listing prints two measures of the same quantity and the pair is what matters. The exp column is the reward the policy earns on average, worked out exactly from its probabilities, and it rises at every checkpoint through 3.28, 3.86, 4.24 and 4.39. The group column estimates that identical quantity from the six responses actually sampled, and it reads 3.65, 5.11, 4.56 and 4.04, overshooting badly at step 10 and undershooting by step 20.
Both columns had to be made to describe the same policy before that comparison meant anything. An earlier version computed exp from the updated probabilities while the group had been sampled from the policy before the update, so the two lagged by a step and part of the gap between them was bookkeeping rather than noise. They now share the same distribution, and everything separating them is sampling. Reporting only the sampled mean, as the original listing did, can make training look like it is going backwards when it is not. Six draws is a small sample, so the group mean scatters widely around the value it stands in for, and on some runs it will rise steadily while on others it lurches around. Whatever it does on a given run, the scatter is real and it enters every single advantage computed from that mean, because the advantage is the reward minus exactly this noisy number.
That is the trade against PPO stated plainly, and trust me when I tell you it can’t get more plain than that. A critic network gives a low variance baseline and costs a second model. A group mean gives a high variance baseline and costs nothing beyond the extra generations, which were needed anyway. Larger groups reduce the variance and cost proportionally more compute per step, which is what exercise 3 asks you to explore at group sizes of 2 and 32. The normalization by standard deviation deserves a note because it connects back to what we found in the reward model. Dividing by the group’s spread makes the advantage scale-free, so a prompt where all responses score near 8 and one where all score near 2 produce advantages of comparable magnitude. Given that the reward model has no absolute scale to begin with, working in relative terms throughout is consistent rather than merely convenient.
A production model like DeepSeek-V3 layers three reward sources on top of this. Rule based rewards handle anything checkable, so a math answer is either right or wrong and no learned model is needed. Model-based rewards handle open ended work where correctness is a judgment. And self rewarding has the model vote on its own outputs against a written set of principles. Mixing sources is more robust than trusting one reward model, for the reason the next section gives.
33.6 Why the Tether Is Not Optional
The KL penalty appears in both algorithms as a term pulling the policy back toward the SFT model it started from. If we look at it in a list of implementation details then it reads like a stability trick, the sort of thing added to stop a training run oscillating. However it is not that. It is the single mechanism standing between a working alignment procedure and a model that games its own reward, and removing it does not produce instability so much as confident nonsense.
The reward model is trained on a finite set of comparisons and it is wrong somewhere, necessarily. A policy optimizing it hard enough will find those places, because finding the maximum of a function is precisely what optimization does, and the maximum of an imperfect approximation sits wherever the approximation is most wrong. The characteristic failure is a policy that produces text scoring extremely well and reading as nonsense.
Consider a reward model that has learned a mild preference for longer answers, because in the training comparisons the more thorough response usually won. That correlation is real in the data and it is not what anyone meant. Without a tether the policy discovers that padding raises the score, and it pads, and it keeps padding, since nothing in the reward model penalizes a response that never stops. If you do happen to work through the exercises, then exercise 4 asks you to build exactly that failure and then fix it.
What the penalty does is make drift cost something. The policy can still move where the reward genuinely improves, and it has to pay for every step away from the SFT model, so it will not travel far into territory where the reward model has never been checked. The coefficient sets the exchange rate, and setting it too high freezes the model while setting it too low returns you to the failure above.
33.7 Key Takeaways
RLHF exists because SFT treats every training example as equally correct and therefore cannot express that one acceptable response is better than another. That needs a score, which is the reinforcement learning setting we covered back in Chapter 28.
The reward model is trained with the Bradley-Terry objective, where the probability A beats B is the sigmoid of the difference in their scores and the loss is the negative log of that for the ordering a person chose.
The bias in a reward model has exactly zero gradient. It appears in both scores and cancels from the difference, so preference data cannot determine it, and the reward model prints it as 0.000 to make the point.
A reward model therefore has no absolute scale. It ranks responses and cannot say whether any of them are good, because adding a constant to every score changes nothing it predicts.
PPO clips the ratio between new and old action probabilities, and the ratio exists because PPO reuses one expensive batch of generations for several gradient steps.
We measured the clip firing 68 times in 160 checks, with the maximum ratio falling from 1.52 to 1.00 as the policy converged. With a single gradient step per batch the ratio would be exactly 1 and the clip would never bind.
PPO needs a critic network for its baseline, typically the same size as the policy. At 671 billion parameters that means two such models, which is why DeepSeek-V3 does not use it.
GRPO replaces the critic with the group itself, computing each response’s advantage as its reward minus the group mean divided by the group standard deviation.
The GRPO listing printed the exact expected reward rising through 3.28, 3.86, 4.24 and 4.39 while the group mean estimating that same quantity read 3.65, 5.11, 4.56 and 4.04, overshooting and then undershooting it.
That gap is the variance a six response group carries. A critic buys a low variance baseline for the price of a second model, and a group buys a noisy one for the price of generations already needed.
Normalizing by the group’s spread is consistent with the reward model having no absolute scale in the first place, so working in relative terms throughout costs nothing.
The KL penalty is not a stability trick, it is what prevents reward hacking. An imperfect reward model is maximized wherever it is most wrong, and the tether makes travelling there cost something.
33.8 Exercises
Add more preference pairs to 161_Reward_Model.c where safety is deliberately low in the preferred response. Does the reward model still learn to value safety?
Implement the KL divergence penalty explicitly in 162_Ppo.c, so KL = sum(ref_prob * log(ref_prob / new_prob)). How does beta (the KL weight) affect the final policy?
In GRPO, what happens with GROUP_SIZE = 2? With GROUP_SIZE = 32? Larger groups give better baseline estimates but cost more compute per step.
Implement a “reward hacking” scenario: define a reward model that gives high scores to very long responses. Show that without the KL penalty, the policy learns to be excessively verbose. With the KL penalty, it stays reasonable.
DeepSeek-V3 uses rule-based rewards for math (check if the final answer is correct). Implement this: the “reward” is 1.0 if the generated answer matches the ground truth, 0.0 otherwise. Train with GRPO.
The advantage formula A_i = (r_i - mean) / std normalizes rewards within each group. What happens if you skip the std normalization and just use (r_i - mean)? When would this fail?