The Vanishing Gradient Problem
Why basic RNNs forget
14.1 What You Will Learn
In Chapter 13 we built a working RNN that counts ones in short sequences, and it worked well enough that you might reasonably expect it to keep working as the sequences grow. It does not. Feed the same network a sequence of fifty or a hundred steps and it forgets the beginning long before it reaches the end. Training longer does not help and neither does tuning the learning rate. What makes this worth a whole chapter is that the failure is not a bug you can hunt down and repair in the code. The architecture itself multiplies the gradient by a number smaller than one at every single time step, and repeated multiplication by a number smaller than one has exactly one destination. In this chapter we make that decay visible in real numbers and trace where in the backward pass it happens. We then work out why the repair has to change the architecture rather than the training procedure. Chapters 15 and 16 build the two architectures that do it.
14.2 The Problem in Words
During backpropagation through time the gradient starts at the last step and travels backward toward the first, and at every step along the way it passes through two operations that can shrink it. The first is the tanh derivative, which reaches its maximum value of 1.0 only when the hidden state sits at exactly zero and falls away quickly on either side. The second is a multiplication by the recurrent weight matrix W_h, which can either shrink or grow the signal depending on how large its entries are. Put those two together and you get a per step factor that is almost always less than one in a trained network. Training drives the hidden states toward the flat ends of the tanh curve, and the derivative there is small.
Figure 14-1 shows where the first of those two shrinking factors comes from. The tanh curve is the one the hidden state passes through, and its derivative is the curve underneath, which peaks at 1 when z is zero and falls away to nothing in both directions. A hidden unit sitting near zero passes the gradient through untouched, and a hidden unit that has saturated toward either extreme passes through almost none of it.
The trouble is that saturation is the normal condition of a trained network rather than a fault. Units settle into confident values away from zero because that is what makes them useful, and the derivative at those values is small.
The consequence compounds. A factor of 0.5 applied ten times leaves you about a thousandth of what you started with. Apply it twenty times and you are down to a millionth, and by fifty applications the result is a number your float cannot usefully distinguish from zero. The early inputs are still there in the forward pass, feeding the hidden state exactly as they always did, but no error signal ever arrives back at them. The network keeps learning from whatever happened recently and stops learning from anything that happened long ago.
14.3 Measuring Gradient Magnitude
Let us train our counting RNN from Chapter 13 on sequences of increasing length and measure how much gradient reaches the first time step.
/* 077_Gradient_Decay.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#define N_HID 4
#define MAX_LEN 60
static float my_tanh(float z)
{
if (z < -20.0f) return -1.0f;
float e = expf(-2.0f * z);
return (1.0f - e) / (1.0f + e);
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
typedef struct {
float W_x[N_HID];
float W_h[N_HID][N_HID];
float b_h[N_HID];
float W_y[N_HID];
float b_y;
}
RNN;
static void forward(const RNN *n,
const float *x, int len,
float h[MAX_LEN + 1][N_HID],
float *y)
{
int t, i, j;
for (i = 0; i < N_HID; i++) h[0][i] = 0;
for (t = 0; t < len; t++) {
for (i = 0; i < N_HID; i++) {
float z = n->b_h[i] + n->W_x[i] * x[t];
for (j = 0; j < N_HID; j++)
z += n->W_h[i][j] * h[t][j];
h[t + 1][i] = my_tanh(z);
}
y[t] = n->b_y;
for (j = 0; j < N_HID; j++)
y[t] += n->W_y[j] * h[t + 1][j];
}
}
/* Backward pass, returning the gradient norm
at each time step */
static void backward_with_norms(
const RNN *n,
const float *x,
const float *tgt,
int len,
const float h[MAX_LEN + 1][N_HID],
const float *y,
float *grad_norms)
{
int t, i, j;
float dh_next[N_HID];
memset(dh_next, 0, sizeof(dh_next));
for (t = len - 1; t >= 0; t--) {
float dy = 2.0f * (y[t] - tgt[t]);
float dh[N_HID], dz[N_HID];
for (i = 0; i < N_HID; i++)
dh[i] = dy * n->W_y[i] + dh_next[i];
for (i = 0; i < N_HID; i++) {
float hv = h[t + 1][i];
dz[i] = dh[i] * (1.0f - hv * hv);
}
/* Compute gradient norm at this step */
float norm = 0.0f;
for (i = 0; i < N_HID; i++)
norm += dz[i] * dz[i];
grad_norms[t] = sqrtf(norm);
/* Propagate to previous step */
memset(dh_next, 0, sizeof(dh_next));
for (j = 0; j < N_HID; j++)
for (i = 0; i < N_HID; i++)
dh_next[j] += dz[i] * n->W_h[i][j];
}
}
int main(void)
{
RNN net;
int i, j;
srand(42);
for (i = 0; i < N_HID; i++) {
net.W_x[i] = randf() * 0.4f - 0.2f;
net.b_h[i] = 0;
net.W_y[i] = randf() * 0.4f - 0.2f;
for (j = 0; j < N_HID; j++)
net.W_h[i][j] = randf() * 0.4f - 0.2f;
}
net.b_y = 0;
/* Test gradient decay across several
sequence lengths */
int lengths[] = { 5, 10, 20, 40 };
int n_lengths = 4;
int li;
printf("Gradient norm at each time step "
"(backward from end):\n\n");
for (li = 0; li < n_lengths; li++) {
int len = lengths[li];
float x[MAX_LEN], tgt[MAX_LEN];
float h[MAX_LEN + 1][N_HID], y[MAX_LEN];
float grad_norms[MAX_LEN];
float count;
int t;
/* Generate a sequence of random bits */
srand(100);
count = 0;
for (t = 0; t < len; t++) {
x[t] = (randf() > 0.5f) ? 1.0f : 0.0f;
count += x[t];
tgt[t] = count;
}
forward(&net, x, len, h, y);
backward_with_norms(&net, x, tgt, len, h, y,
grad_norms);
printf(" Length %d:\n", len);
printf(" Last step (t=%d): grad_norm = "
"%.6f\n",
len - 1, grad_norms[len - 1]);
printf(" Mid step (t=%d): grad_norm = "
"%.6f\n",
len / 2, grad_norms[len / 2]);
printf(" First step (t=0): grad_norm = "
"%.6f\n",
grad_norms[0]);
if (grad_norms[len - 1] > 0)
printf(" Ratio first/last: %.8f\n",
grad_norms[0] / grad_norms[len - 1]);
printf("\n");
}
return 0;
}
Watch the ratio of first-step gradient to last-step gradient. For short sequences (length 5), the ratio is maybe 0.1 or 0.01. For length 40, it is essentially zero. The gradient has vanished. The first input has no influence on the weight updates. The network cannot learn to use information from early in the sequence.
14.4 The Math of Vanishing
Let us trace the arithmetic precisely rather than describing it. At every time step during the backward pass the gradient passes through the tanh derivative, which multiplies it by (1 - h_t^2). It then passes through the recurrent matrix, which multiplies it by W_h transposed. Running that all the way from step T back to step 0 gives a product with one term per step.
Figure 14-2 works that product out for a range of hidden values, using the numbers the second program in this chapter prints. Read along the h = 0.90 row, which is an entirely ordinary value for a trained unit. The per step factor is 0.19, which sounds survivable, and after twenty steps it has become 3.76 multiplied by ten to the minus fifteen. A float cannot represent the difference between that and zero in any way the optimizer can use.
The h = 0.99 row is worse and the h = 0.00 row is the only one that survives, which is the whole difficulty in one table. The only hidden value that preserves the gradient is the one carrying no information.
Every (1 - h_t^2) term in that chain sits somewhere between 0 and 1, so every one of them shrinks the gradient. The W_h terms are the only thing that could push back. Whether they do depends on the spectral norm of the matrix, and for a randomly initialized or normally trained network that norm is not large enough to compensate. Let us compute the shrinking factor directly across the range of hidden state values a real network produces.
/* 078_Factor.c */
#include <stdio.h>
#include <math.h>
int main(void)
{
/* Simulate the per-step shrinkage factor */
/* tanh derivative at various hidden state values */
float h_values[] = { 0.0f, 0.3f, 0.5f, 0.7f,
0.9f, 0.95f, 0.99f };
int n = 7;
int i;
printf("Tanh derivative (1 - h^2) at various "
"hidden values:\n\n");
printf(" h tanh' after 10 "
"after 20 after 50\n");
printf(" ------- ------ -------- "
"-------- --------\n");
for (i = 0; i < n; i++) {
float h = h_values[i];
float d = 1.0f - h * h;
double d10 = pow(d, 10);
double d20 = pow(d, 20);
double d50 = pow(d, 50);
printf(" %5.2f %6.4f"
" %.2e %.2e %.2e\n",
h, d, d10, d20, d50);
}
printf("\nWhen h is near 0, the derivative is 1 "
"and gradients survive.\n");
printf("When h is 0.9 (common after training), "
"the derivative is 0.19.\n");
printf("After 20 steps of 0.19: %.2e "
"(effectively zero).\n",
pow(0.19, 20));
printf("\nThis is why basic RNNs cannot learn "
"long-range dependencies.\n");
printf("The gradient vanishes exponentially "
"with sequence length.\n");
return 0;
}
Read the table one row at a time and the exponential nature of the problem becomes hard to miss. The first row is the best case the architecture can offer, where a hidden state of exactly 0.00 gives a derivative of 1.0000 and the gradient survives fifty steps completely intact at 1.00e+00. That row is also the one you will never see in practice. A hidden state pinned at zero carries no information. Move down to h = 0.30 and the derivative drops to 0.9100, which still looks harmless. Follow that row out to fifty steps and you find 8.96e-03. Roughly one part in a hundred of the original signal survives, and 0.30 is a very quiet hidden state.
The row that matters most is h = 0.90, because that is what a trained tanh unit actually settles at once it has learned to represent something firmly. The derivative there is 0.1900, and twenty steps of 0.1900 gives 3.76e-15. That number is smaller than the spacing between adjacent floats near 1.0. The gradient has not merely become small, it has fallen below the resolution of the arithmetic carrying it. Fifty steps gives 8.66e-37, and the last row shows that a confidently saturated unit at h = 0.99 reaches 8.76e-86 over the same distance. There is no learning rate that rescues a number like that.
Notice what the table does not contain, because the absence is the point. Nowhere in it does any column grow. Every path from left to right shrinks, and every row shrinks faster than the row above it. The only row that holds its value is the one where the unit has stopped representing anything. The vanishing gradient is not an unlucky interaction between components that could be tuned apart. It falls directly out of multiplying a bounded derivative by itself once per time step, and the basic RNN gives the gradient no other route to travel.
14.5 Long-Range Dependency Failure
Let us build a task that specifically requires remembering the first input and demonstrate that the RNN fails.
The task. the first bit of the sequence determines the target. If the first bit is 1, the target is the sum of all bits. If the first bit is 0, the target is negative sum. The network must remember the first bit throughout the entire sequence to get the sign right.
/* 079_Long_Range.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#define N_HID 8
#define MAX_LEN 60
static float my_tanh(float z)
{
if (z < -20) return -1;
float e = expf(-2 * z);
return (1 - e) / (1 + e);
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
typedef struct {
float W_x[N_HID];
float W_h[N_HID][N_HID];
float b_h[N_HID];
float W_y[N_HID];
float b_y;
}
RNN;
static float run_seq(const RNN *n,
const float *x, int len)
{
float h[N_HID] = {0};
float h_new[N_HID];
int t, i, j;
for (t = 0; t < len; t++) {
for (i = 0; i < N_HID; i++) {
float z = n->b_h[i] + n->W_x[i] * x[t];
for (j = 0; j < N_HID; j++)
z += n->W_h[i][j] * h[j];
h_new[i] = my_tanh(z);
}
for (i = 0; i < N_HID; i++) h[i] = h_new[i];
}
float y = n->b_y;
for (i = 0; i < N_HID; i++) y += n->W_y[i] * h[i];
return y;
}
static void train_step(RNN *n, const float *x,
int len, float target, float lr)
{
/* Full BPTT, simplified to use numerical
gradients for clarity */
float *w = (float *)n;
int nw = sizeof(RNN) / sizeof(float);
int i;
float h = 0.001f;
float y0 = run_seq(n, x, len);
float loss0 = (y0 - target) * (y0 - target);
for (i = 0; i < nw; i++) {
float orig = w[i];
w[i] = orig + h;
float y1 = run_seq(n, x, len);
float loss1 = (y1 - target) * (y1 - target);
float grad = (loss1 - loss0) / h;
w[i] = orig - lr * grad;
}
}
int main(void)
{
RNN net;
int i, j;
srand(42);
for (i = 0; i < N_HID; i++) {
net.W_x[i] = randf() * 0.2f - 0.1f;
net.b_h[i] = 0;
net.W_y[i] = randf() * 0.2f - 0.1f;
for (j = 0; j < N_HID; j++)
net.W_h[i][j] = randf() * 0.2f - 0.1f;
}
net.b_y = 0;
printf("Long-range dependency test:\n");
printf(" First bit determines the sign of "
"the output.\n");
printf(" The network must remember the first "
"bit through\n");
printf(" the entire sequence.\n\n");
int test_lens[] = { 5, 10, 20 };
int n_tests = 3;
int li;
for (li = 0; li < n_tests; li++) {
int len = test_lens[li];
/* Reset network */
srand(42);
for (i = 0; i < N_HID; i++) {
net.W_x[i] = randf() * 0.2f - 0.1f;
net.b_h[i] = 0;
net.W_y[i] = randf() * 0.2f - 0.1f;
for (j = 0; j < N_HID; j++)
net.W_h[i][j] = randf() * 0.2f - 0.1f;
}
net.b_y = 0;
/* Train */
int epoch;
for (epoch = 0; epoch < 200; epoch++) {
int s;
for (s = 0; s < 20; s++) {
float x[MAX_LEN];
int t;
float count = 0;
/* Generate random sequence */
for (t = 0; t < len; t++) {
x[t] = (randf() > 0.5f)
? 1.0f
: 0.0f;
count += x[t];
}
/* Positive count when the first bit
is 1, negative otherwise */
float target =
(x[0] > 0.5f) ? count : -count;
train_step(&net, x, len,
target, 0.001f);
}
}
/* Test: does it get the sign right? */
int correct = 0, total = 50;
for (i = 0; i < total; i++) {
float x[MAX_LEN];
int t;
float count = 0;
for (t = 0; t < len; t++) {
x[t] = (randf() > 0.5f) ? 1.0f : 0.0f;
count += x[t];
}
float target = (x[0] > 0.5f)
? count
: -count;
float y = run_seq(&net, x, len);
/* Check if sign matches */
if ((y > 0 && target > 0) ||
(y < 0 && target < 0) ||
(y == 0 && target == 0))
correct++;
}
printf(" Length %2d: sign accuracy = "
"%d/%d (%.0f%%)\n",
len, correct, total,
100.0f * correct / total);
}
printf("\n At short lengths, the RNN can "
"remember the first bit.\n");
printf(" As length increases, accuracy drops "
"toward 50%% (random).\n");
printf(" The gradient from the first step "
"has vanished.\n");
return 0;
}
At length 5, the RNN should get most signs right because the gradient can reach the first step. At length 20, accuracy should drop toward 50% (random guessing) because the gradient has vanished and the first bit is forgotten.
14.6 Visualizing the Gradient Flow
The table in the previous section shows what happens to a single scalar factor. Let us watch it happen inside a real network. This program prints the gradient norm at every one of thirty time steps, with a bar of hashes beside each number so the shape is visible at a glance.
/* 080_Gradient_Flow.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#define N_HID 4
#define SEQ_LEN 30
static float my_tanh(float z)
{
if (z < -20) return -1;
float e = expf(-2 * z);
return (1 - e) / (1 + e);
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
typedef struct {
float W_x[N_HID];
float W_h[N_HID][N_HID];
float b_h[N_HID];
float W_y[N_HID];
float b_y;
}
RNN;
int main(void)
{
RNN net;
int i, j, t;
srand(42);
for (i = 0; i < N_HID; i++) {
net.W_x[i] = randf() * 0.4f - 0.2f;
net.b_h[i] = 0;
net.W_y[i] = randf() * 0.4f - 0.2f;
for (j = 0; j < N_HID; j++)
net.W_h[i][j] = randf() * 0.4f - 0.2f;
}
net.b_y = 0;
/* Generate sequence */
float x[SEQ_LEN], tgt[SEQ_LEN];
float count = 0;
srand(100);
for (t = 0; t < SEQ_LEN; t++) {
x[t] = (randf() > 0.5f) ? 1.0f : 0.0f;
count += x[t];
tgt[t] = count;
}
/* Forward pass */
float h[SEQ_LEN + 1][N_HID], y[SEQ_LEN];
for (i = 0; i < N_HID; i++) h[0][i] = 0;
for (t = 0; t < SEQ_LEN; t++) {
for (i = 0; i < N_HID; i++) {
float z = net.b_h[i] + net.W_x[i] * x[t];
for (j = 0; j < N_HID; j++)
z += net.W_h[i][j] * h[t][j];
h[t+1][i] = my_tanh(z);
}
y[t] = net.b_y;
for (j = 0; j < N_HID; j++)
y[t] += net.W_y[j] * h[t+1][j];
}
/* Backward pass with gradient norm at each step */
float dh_next[N_HID];
memset(dh_next, 0, sizeof(dh_next));
float norms[SEQ_LEN];
for (t = SEQ_LEN - 1; t >= 0; t--) {
float dy = 2.0f * (y[t] - tgt[t]);
float dh[N_HID], dz[N_HID];
for (i = 0; i < N_HID; i++)
dh[i] = dy * net.W_y[i] + dh_next[i];
for (i = 0; i < N_HID; i++) {
float hv = h[t+1][i];
dz[i] = dh[i] * (1 - hv * hv);
}
float norm = 0;
for (i = 0; i < N_HID; i++)
norm += dz[i] * dz[i];
norms[t] = sqrtf(norm);
memset(dh_next, 0, sizeof(dh_next));
for (j = 0; j < N_HID; j++)
for (i = 0; i < N_HID; i++)
dh_next[j] += dz[i] * net.W_h[i][j];
}
/* Print as a bar chart */
printf("Gradient norm at each time step "
"(length %d):\n\n", SEQ_LEN);
printf(" t norm bar\n");
float max_norm = 0;
for (t = 0; t < SEQ_LEN; t++)
if (norms[t] > max_norm) max_norm = norms[t];
for (t = SEQ_LEN - 1; t >= 0; t--) {
int bar_len = (max_norm > 0)
? (int)(norms[t] / max_norm * 40) : 0;
printf(" %2d %.6f ", t, norms[t]);
for (i = 0; i < bar_len; i++) printf("#");
printf("\n");
}
printf("\n The gradient is strong at the end "
"and vanishes\n");
printf(" toward the beginning. Early inputs "
"get almost\n");
printf(" no learning signal.\n");
return 0;
}
Start at the top of the output where t = 29, the final step, and the norm is 6.609724 with a bar running the full width of the display. Walk down through the twenties and the decline is gentle and slightly noisy. The norm falls from 5.068781 at t = 28 to 3.583501 at t = 20, which is roughly half across ten steps. Nothing there looks alarming. Train only on sequences of ten or fifteen steps and you would never suspect the architecture had a problem. That is precisely why this one catches people out.
Keep walking down and the character of the decline changes. Between t = 10 and t = 5 the norm drops from 1.204731 to 0.101964, losing an order of magnitude across five steps rather than half across ten. From t = 5 to t = 0 it falls from 0.101964 to 0.000032, which is another three orders of magnitude across the same five steps. The bars stop rendering entirely at t = 5 because there is no longer enough magnitude left to fill a single character position. The final step carries a gradient roughly two hundred thousand times larger than the first one.
That accelerating shape is worth sitting with, because it explains a training behavior you may have already met without recognizing it. A network in this state will show a loss curve that improves quickly, plateaus at a mediocre value, and then refuses to improve further no matter how long you leave it running. The last handful of steps have absorbed all the available learning signal. Every parameter governing anything earlier has effectively frozen. Information still flows forward through the hidden state without any trouble, since the forward pass does no shrinking at all. Only the error flowing backward dies, and a network that cannot receive error cannot learn.
14.7 A Highway for Gradients
Everything above traces back to a single structural fact. The gradient has exactly one road to travel, and that road passes through a tanh derivative and a matrix multiply at every step. Fix the road and the problem disappears. Suppose some path existed along which the gradient could move from one step to the next without being multiplied by anything smaller than one. Distance along the sequence would stop costing magnitude, and a signal leaving step zero would arrive at step fifty in roughly the shape it started.
That is exactly the move LSTM makes in Chapter 15. It adds a second state vector, the cell state, and connects it from one step to the next by addition rather than by a squashing function and a matrix. Addition has a derivative of one, so gradient flowing along the cell state is not attenuated by the passage of time at all. Gates built from sigmoid units decide what gets written into that state and what gets read back out. The state itself stays a clear channel, because the gates sit beside it rather than across it. GRU in Chapter 16 reaches a similar destination with fewer parameters by merging the gates and folding the cell state back into the hidden state.
Neither architecture makes gradients bigger, which is worth being precise about because the intuitive fix people reach for first is amplification. Amplification gives you the exploding gradient problem instead. That one is easier to manage with clipping, but it is still a problem. What LSTM and GRU do is remove the obstruction rather than push harder against it.
14.8 Key Takeaways
The vanishing gradient problem is exponential decay during backpropagation through time, where each step multiplies the gradient by the tanh derivative and by the recurrent matrix W_h. The derivative is at most 1.0 and typically falls between 0.1 and 0.7 in a trained network.
After twenty to fifty steps the gradient arriving at the earliest inputs is numerically indistinguishable from zero, so the network cannot learn any dependency spanning that distance.
A hidden state of 0.90 gives a per step factor of 0.19, which reaches 3.76e-15 after twenty steps. A saturated state of 0.99 reaches 8.76e-86 over fifty steps.
The decay accelerates rather than proceeding evenly. Gradient norms fall by about half across the last ten steps of a thirty step sequence and then by five orders of magnitude across the first ten.
This is a property of the architecture and not a defect in any implementation, so no amount of training time or hyperparameter search repairs it for a basic RNN.
The exploding gradient problem is the same mechanism running in the opposite direction, where W_h amplifies instead of attenuating, and gradient clipping from Chapter 13 handles it adequately.
The repair has to be architectural, and it works by giving the gradient a path that multiplies by one rather than by making the existing path steeper. LSTM in Chapter 15 and GRU in Chapter 16 both take this route.
Deep feedforward networks suffer the same decay, but RNNs have it worse because the same W_h is applied once per time step. A feedforward network gives each layer its own weights, which spreads the per layer factors around instead of raising one number to a power.
14.9 Exercises
Run 077_Gradient_Decay.c with N_HID = 16 instead of 4. Does the gradient vanish faster or slower? Why?
Initialize W_h as an identity matrix scaled by 0.99 (net.W_h[i][j] = 0.99 * (i == j)). Does this help the vanishing gradient problem? What about scaling by 1.01?
Modify 080_Gradient_Flow.c to print gradients after training for 1000 epochs. Do the trained gradients vanish more or less than the untrained ones?
Compute how many steps it takes for the gradient to drop below 1e-7 with a per-step factor of 0.5, 0.7, and 0.9.
Try using ReLU instead of tanh in the RNN. What happens to the hidden state values after 30 steps? Does the gradient vanish or explode?
Research the problem that motivates LSTM: the “minimal time lag” between an input event and when the output needs it. What is the longest time lag your basic RNN can handle at 90% accuracy?