Sequence-to-Sequence
Encoder-decoder for translation and beyond
18.1 What You Will Learn
Every architecture so far produces either one output per input step, which is the sequence labelling of Chapter 17, or one output for the whole sequence, which is classification. Both share an assumption that turns out to be restrictive, namely that the shape of the output is fixed by the shape of the input. Translation breaks it immediately. “I love cats” is three words in and three words out, which looks fine until “I am a student” goes in as four and comes out as “Je suis etudiant”, which is three. Nothing in the architecture knows how to drop a word.
The encoder-decoder answer is to split the problem in half. One network reads the entire input and compresses it into a single fixed-size vector, and a second network reads that vector and generates output tokens one at a time until it decides to stop. The input length and the output length are then independent of each other, because the only thing passing between the two halves is a vector whose size does not depend on either.
That single vector is also the architecture’s weakness, and this chapter builds up to measuring it. We construct the encoder, then the decoder, then look at why training needs teacher forcing and what that costs, then join the halves into a working pipeline, and finally test how much of the input actually survives the compression. The answer to that last question is what motivates attention in Chapter 19, so the bottleneck measurement is where the chapter is heading from the first page.
18.2 The Problem
Three translation pairs make the mismatch concrete, and none of them is unusual.
| Input | Output | Length change |
|---|---|---|
| hello | bonjour | 5 to 7 |
| good morning | bonjour | 12 to 7 |
| cat | chat | 3 to 4 |
The second row is the interesting one because it grows shorter, and the first grows longer, so there is no fixed ratio to exploit either. An RNN that emits an output at every step produces exactly as many outputs as it consumed inputs, which handles none of these. Padding the shorter side does not fix it, since the model would then have to learn to emit padding in the right places, and the alignment between input and output positions is not one to one in the first place. The word bonjour corresponds to a whole greeting rather than to any particular character of it.
Translation is the obvious case, but summarization compresses a long input into a short one by design, and question answering maps a passage plus a question onto an answer of whatever length the answer happens to need. All three want the output length to be decided by the model rather than by the input.
Figure 18-1 shows the arrangement that solves it, drawn on the task the programs in this chapter actually run, which reverses a short sequence rather than translating. The encoder reads all three input tokens and emits nothing at all, which is the part worth pausing on. Its outputs are discarded and only the hidden state it finishes with is kept.
That single vector is handed to the decoder as its starting state, and from there the decoder runs on its own clock. It is fed a start token, produces one output, is fed that output back, and repeats until it emits an end token. Nothing ties the number of decoder steps to the number of encoder steps, which is precisely the constraint that needed breaking.
The cost is visible in the same picture. Everything the decoder will ever know about the input has to fit through that one arrow, and we measure what that costs as the input grows.
18.3 The Encoder
The encoder is the simpler half and there is genuinely nothing new in it. Run a recurrent network over the input, ignore every output it produces along the way, and keep only the hidden state left at the end. That final state is the context vector, and by construction it is the only thing the decoder will ever see of the input.
/* 096_Encoder.c */
#include <stdio.h>
#include <math.h>
static float my_tanh(float z)
{
if (z < -20) return -1;
float e = expf(-2 * z);
return (1 - e) / (1 + e);
}
#define N_HID 4
static void gru_step(const float wx[N_HID],
const float wh[N_HID][N_HID],
const float bh[N_HID], float x,
const float hp[N_HID],
float hn[N_HID])
{
/* Simplified to a tanh RNN for clarity. The
principle is the same with a GRU. */
int i, j;
for (i = 0; i < N_HID; i++) {
float z = bh[i] + wx[i] * x;
for (j = 0; j < N_HID; j++)
z += wh[i][j] * hp[j];
hn[i] = my_tanh(z);
}
}
int main(void)
{
/* Encode "hello" as normalized character IDs */
/* h, e, l, l, o */
float input[] = { 0.8f, 0.5f, 1.2f, 1.2f, 1.5f };
int input_len = 5;
float wx[N_HID] = { 0.5f, -0.3f, 0.2f, 0.4f };
float wh[N_HID][N_HID] = {
{ 0.3f, 0.1f, 0, 0 },
{ 0, 0.3f, 0.1f, 0 },
{ 0, 0, 0.3f, 0.1f },
{ 0.1f, 0, 0, 0.3f },
};
float bh[N_HID] = { 0, 0, 0, 0 };
float h[N_HID] = { 0, 0, 0, 0 };
float h_new[N_HID];
int t, i;
printf("Encoder: reading input sequence\n\n");
printf(" t input hidden state\n");
for (t = 0; t < input_len; t++) {
gru_step(wx, wh, bh, input[t], h, h_new);
printf(" %d %.1f "
"[%+.3f, %+.3f, %+.3f, %+.3f]\n",
t, input[t], h_new[0], h_new[1],
h_new[2], h_new[3]);
for (i = 0; i < N_HID; i++) h[i] = h_new[i];
}
printf("\nContext vector (final hidden state):\n");
printf(" [%+.3f, %+.3f, %+.3f, %+.3f]\n",
h[0], h[1], h[2], h[3]);
printf("\nThis single vector must capture "
"everything about\n");
printf("\"hello\" that the decoder needs to "
"produce \"bonjour\".\n");
return 0;
}
Figure 18-2 accumulates a context vector over the input. Follow the hidden state down the table and watch it accumulate. After the first input of 0.8 the state is [+0.380, −0.235, +0.159, +0.310], and by the last input of 1.5 it has grown to [+0.715, −0.491, +0.439, +0.688]. Every component has moved in the same direction it started, which is what you would expect from a sequence of positive inputs fed through a network whose weights do not change sign.
The last row is the context vector and it deserves a moment of skepticism. Four floats now stand in for the entire input, and they will stand in for it during every step of decoding. Whatever the decoder needs to know about “hello” in order to produce “bonjour” has to be recoverable from those four numbers, because nothing else crosses the boundary. Increase the hidden size to 256 and you have 256 floats instead, which is more room but still a fixed amount that does not grow when the input does.
For a five character input this is comfortable. The question the rest of the chapter builds toward is what happens at fifty characters, or five hundred, when the same four numbers have to carry proportionally more, and the last section of this chapter measures the answer rather than guessing at it.
18.4 The Decoder
The decoder is another recurrent network with two differences from anything built so far. It starts from the context vector rather than from zeros, which is how the input reaches it, and its input at each step is the token it produced at the previous step rather than anything external.
/* 097_Decoder.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
static float my_tanh(float z)
{
if (z < -20) return -1;
float e = expf(-2 * z);
return (1 - e) / (1 + e);
}
#define N_HID 4
#define VOCAB_SIZE 6 /* simplified output vocabulary */
static void rnn_step(const float wx[N_HID],
const float wh[N_HID][N_HID],
const float bh[N_HID], float x,
const float hp[N_HID],
float hn[N_HID])
{
int i, j;
for (i = 0; i < N_HID; i++) {
float z = bh[i] + wx[i] * x;
for (j = 0; j < N_HID; j++)
z += wh[i][j] * hp[j];
hn[i] = my_tanh(z);
}
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
int main(void)
{
/* This context vector came from encoding
"hello" in the previous step */
float context[N_HID] = { 0.45f, -0.23f,
0.61f, 0.18f };
/* Decoder weights */
float wx[N_HID] = { 0.4f, -0.2f, 0.3f, 0.1f };
float wh[N_HID][N_HID] = {
{ 0.3f, 0.1f, 0, 0 },
{ 0, 0.3f, 0.1f, 0 },
{ 0, 0, 0.3f, 0.1f },
{ 0.1f, 0, 0, 0.3f },
};
float bh[N_HID] = { 0, 0, 0, 0 };
/* Output projection: hidden -> vocabulary */
float W_out[VOCAB_SIZE][N_HID] = {
/* token 0, <START> */
{ 0.5f, -0.1f, 0.3f, 0.2f },
/* token 1: 'b' */
{ -0.3f, 0.4f, 0.1f, -0.2f },
/* token 2: 'o' */
{ 0.2f, 0.3f, -0.4f, 0.1f },
/* token 3: 'n' */
{ -0.1f, -0.2f, 0.5f, 0.3f },
/* token 4: 'j' */
{ 0.3f, 0.1f, 0.1f, -0.4f },
/* token 5: <END> */
{ 0.1f, 0.2f, 0.1f, 0.5f },
};
const char *vocab[] = { "<START>", "b", "o",
"n", "j", "<END>" };
float h[N_HID], h_new[N_HID];
float input_token;
int t, i, j;
/* Start the decoder from the context vector */
for (i = 0; i < N_HID; i++) h[i] = context[i];
printf("Decoder: generating output sequence\n\n");
printf(" t input_tok hidden "
"probs pred\n");
/* First input is always <START>, token 0 */
input_token = 0.0f;
for (t = 0; t < 6; t++) {
/* RNN step */
rnn_step(wx, wh, bh, input_token, h, h_new);
/* Project to vocabulary */
float logits[VOCAB_SIZE];
for (i = 0; i < VOCAB_SIZE; i++) {
logits[i] = 0;
for (j = 0; j < N_HID; j++)
logits[i] += W_out[i][j] * h_new[j];
}
softmax(logits, VOCAB_SIZE);
/* Find prediction */
int pred = 0;
for (i = 1; i < VOCAB_SIZE; i++)
if (logits[i] > logits[pred]) pred = i;
printf(" %d %-8s "
"[%+.2f,%+.2f,%+.2f,%+.2f] ", t,
t == 0 ? "<START>" : vocab[pred],
h_new[0], h_new[1], h_new[2], h_new[3]);
for (i = 0; i < VOCAB_SIZE; i++)
printf("%.2f ", logits[i]);
printf(" -> %s\n", vocab[pred]);
/* Feed prediction as next input
(autoregressive) */
input_token = (float)pred / VOCAB_SIZE;
for (i = 0; i < N_HID; i++) h[i] = h_new[i];
/* Stop if <END> predicted */
if (pred == 5) break;
}
printf("\nThe decoder generates tokens one at "
"a time.\n");
printf("Each token feeds back as input to the "
"next step.\n");
printf("Generation stops when "
"<END> is predicted.\n");
return 0;
}
Figure 18-3 runs an untrained decoder autoregressively. The table shows the mechanism working and the model failing, which is the usual state of an untrained example. Step 0 starts from the context vector, produces a distribution over the six token vocabulary, and takes the argmax, which comes out as . Step 1 then feeds that predicted back in as input, which is what the input_tok column is showing, and predicts again. The loop is closed and running correctly even though what it is producing is nonsense.
Watch the hidden state column while this happens. It begins at [+0.11, −0.01, +0.20, +0.10] and shrinks at every step to [+0.03, +0.02, +0.07, +0.04], then [+0.01, +0.01, +0.02, +0.02], and is effectively zero by step 4. The probabilities track it exactly, starting spread between 0.15 and 0.18 and flattening to a uniform 0.17 across all six tokens once the state has died. An untrained decoder with small random weights has nothing to sustain its state, so it forgets the context vector within four steps and then predicts uniformly forever.
That decay matters more here than it did in earlier chapters. In a labelling task a dead hidden state costs you accuracy at the later positions. In an autoregressive decoder it costs you everything after the first mistake, because each output becomes the next input and there is no external signal arriving to correct the trajectory. One wrong token and the model is conditioning on its own error for the rest of the sequence, which is exactly the problem the next section addresses.
18.5 Teacher Forcing
The fix for a decoder that compounds its own errors is to stop letting it, at least while training. Instead of feeding the token the model just predicted, feed the token the target says should have been there. The model still makes its prediction and is still scored on it, but the input to the next step comes from the answer key rather than from the model.
This listing runs both regimes over the same decoder so the difference is measured rather than asserted. The decoder here is a lookup table rather than a trained network, which keeps it deterministic and lets one deliberate flaw be planted in a known place. Every row of the table picks the right next token except the row for ‘b’, where the model wrongly prefers ‘j’ over ‘o’.
/* 098_Teacher_Forcing.c */
#include <stdio.h>
#define V 6 /* vocabulary size */
/* 0=<S> 1='b' 2='o' 3='n' 4='j' 5=<E> */
static const char *tok[V] = {
"<S>", "b", "o", "n", "j", "<E>"
};
/* A stand-in for a partly trained decoder. Row p
holds the score the model gives each next token
after p. Every row is right except row 1, where
the model wrongly prefers 'j' over 'o'. */
static const float score[V][V] = {
/* <S> b o n j <E> */
/* <S> */ { 0.0f, 0.7f, 0.1f, 0.1f, 0.0f, 0.1f },
/* b */ { 0.0f, 0.0f, 0.3f, 0.1f, 0.5f, 0.1f },
/* o */ { 0.0f, 0.1f, 0.0f, 0.6f, 0.2f, 0.1f },
/* n */ { 0.0f, 0.0f, 0.1f, 0.0f, 0.8f, 0.1f },
/* j */ { 0.0f, 0.0f, 0.1f, 0.0f, 0.0f, 0.9f },
/* <E> */ { 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 1.0f },
};
static int predict(int prev)
{
int best = 0, i;
for (i = 1; i < V; i++)
if (score[prev][i] > score[prev][best])
best = i;
return best;
}
int main(void)
{
/* The target output is b o n j <E> */
int target[] = { 1, 2, 3, 4, 5 };
int len = 5;
int t, prev, pred, hits;
printf("Target sequence: ");
for (t = 0; t < len; t++)
printf("%s ", tok[target[t]]);
printf("\n\n");
printf("Autoregressive, feeding back "
"predictions\n\n");
printf(" step input predict target \n");
printf(" ---- ----- ------- ------ \n");
prev = 0;
hits = 0;
for (t = 0; t < len; t++) {
pred = predict(prev);
if (pred == target[t]) hits++;
printf(" %3d %-5s %-7s %-6s %s\n",
t, tok[prev], tok[pred], tok[target[t]],
pred == target[t] ? "ok" : "wrong");
/* feed our own output back */
prev = pred;
if (pred == 5) {
t++;
break;
}
}
printf("\n correct: %d of %d\n", hits, len);
printf(" One wrong step at t=1 sent it to <E>\n");
printf(" early, so the rest was never tried.\n\n");
printf("Teacher forcing, feeding back the "
"target\n\n");
printf(" step input predict target \n");
printf(" ---- ----- ------- ------ \n");
prev = 0;
hits = 0;
for (t = 0; t < len; t++) {
pred = predict(prev);
if (pred == target[t]) hits++;
printf(" %3d %-5s %-7s %-6s %s\n",
t, tok[prev], tok[pred], tok[target[t]],
pred == target[t] ? "ok" : "wrong");
/* feed the correct token */
prev = target[t];
}
printf("\n correct: %d of %d\n", hits, len);
printf(" The same flawed model scores far "
"better,\n");
printf(" because one bad step no longer spoils\n");
printf(" the input to every step after it.\n\n");
printf("The model is identical in both runs.\n");
printf("Only the input at each step differs. At\n");
printf("training time we can feed "
"the target, at\n");
printf("inference we cannot. That gap is called\n");
printf("exposure bias.\n");
return 0;
}
Figure 18-4 runs the same flawed decoder both ways. The autoregressive run gets one of five and stops after three steps. Step 0 correctly predicts ‘b’ from . Step 1 feeds that ‘b’ back in, hits the flawed row, and predicts ‘j’ where the target says ‘o’. Step 2 then feeds ‘j’ back in, and the row for ‘j’ points at with a score of 0.9, so the decoder ends the sequence three tokens early. The targets ‘n’ and ‘j’ were never even attempted, which is why the count is 1 rather than 2.
The teacher forced run over the identical table gets four of five. Step 1 makes exactly the same mistake, predicting ‘j’ when the target is ‘o’, and it is scored as wrong. The difference is what happens next. Step 2 receives ‘o’ from the target rather than ‘j’ from the model, lands on the correct row, and predicts ‘n’. Steps 3 and 4 follow correctly for the same reason. One flaw stays one flaw instead of becoming four.
The model is byte for byte identical in both runs, which is the point worth holding onto. Nothing was improved between them. Only the input at each step changed, and the score went from one to four. That is the case for teacher forcing during training, and it is also the whole of the problem with it, because at inference time there is no target sequence to feed. The model trains in a world where its input is always correct and then runs in a world where its input is whatever it just produced. The gap between the two is called exposure bias, and it shows up as models that behave well on sequences resembling their training data and degrade on longer or unfamiliar ones.
18.6 The Full Encoder-Decoder
Both halves now go into one program on a task small enough to verify by eye, which is reversing a three character sequence so that a, b, c becomes C, B, A. The input and output vocabularies are deliberately different, since a real translation model does not share a vocabulary between languages either.
/* 099_Seq2seq.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#include <stdlib.h>
static float my_tanh(float z)
{
if (z < -20) return -1;
float e = expf(-2 * z);
return (1 - e) / (1 + e);
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N_HID 6
#define VOCAB_IN 5
#define VOCAB_OUT 5
#define MAX_LEN 10
static void rnn_step(const float wx[N_HID],
const float wh[N_HID][N_HID],
const float bh[N_HID], float x,
const float hp[N_HID],
float hn[N_HID])
{
int i, j;
for (i = 0; i < N_HID; i++) {
float z = bh[i] + wx[i] * x;
for (j = 0; j < N_HID; j++)
z += wh[i][j] * hp[j];
hn[i] = my_tanh(z);
}
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
int main(void)
{
/* Input vocab 0=<PAD> 1='a' 2='b' 3='c' 4='d' */
/* Output vocab 0=<START> 1='A' 2='B' 3='C'
4 = <END> */
const char *in_vocab[] = { "<PAD>", "a", "b",
"c", "d" };
const char *out_vocab[] = { "<START>", "A", "B",
"C", "<END>" };
/* Task: reverse the input sequence */
/* Input: a b c (tokens: 1 2 3) */
/* Output: C B A (tokens: 3 2 1 then <END>=4) */
int input_tokens[] = { 1, 2, 3 };
int target_tokens[] = { 3, 2, 1, 4 };
/* C B A <END> */
int input_len = 3;
int target_len = 4;
/* Encoder and decoder weights (random) */
float enc_wx[N_HID], enc_bh[N_HID];
float enc_wh[N_HID][N_HID];
float dec_wx[N_HID], dec_bh[N_HID];
float dec_wh[N_HID][N_HID];
float W_out[VOCAB_OUT][N_HID];
int i, j;
srand(42);
for (i = 0; i < N_HID; i++) {
enc_wx[i] = randf()*0.4f-0.2f;
enc_bh[i] = 0;
dec_wx[i] = randf()*0.4f-0.2f;
dec_bh[i] = 0;
for (j = 0; j < N_HID; j++) {
enc_wh[i][j] = randf()*0.4f-0.2f;
dec_wh[i][j] = randf()*0.4f-0.2f;
}
}
for (i = 0; i < VOCAB_OUT; i++)
for (j = 0; j < N_HID; j++)
W_out[i][j] = randf()*0.4f-0.2f;
/* === ENCODE === */
float h[N_HID] = {0}, h_new[N_HID];
printf("=== ENCODER ===\n");
for (int t = 0; t < input_len; t++) {
float x = (float)input_tokens[t] / VOCAB_IN;
rnn_step(enc_wx, enc_wh, enc_bh, x, h, h_new);
printf(" t=%d input='%s' h=[", t,
in_vocab[input_tokens[t]]);
for (i = 0; i < N_HID; i++)
printf("%+.2f%s", h_new[i],
i<N_HID-1?",":"");
printf("]\n");
for (i = 0; i < N_HID; i++) h[i] = h_new[i];
}
printf("\nContext vector: [");
for (i = 0; i < N_HID; i++)
printf("%+.3f%s", h[i], i<N_HID-1?", ":"");
printf("]\n");
/* === DECODE (with teacher forcing) === */
printf("\n=== DECODER (teacher forcing) ===\n");
float dec_input = 0.0f; /* <START> = token 0 */
for (int t = 0; t < target_len; t++) {
rnn_step(dec_wx, dec_wh, dec_bh, dec_input,
h, h_new);
/* Project to output vocabulary */
float logits[VOCAB_OUT];
for (i = 0; i < VOCAB_OUT; i++) {
logits[i] = 0;
for (j = 0; j < N_HID; j++)
logits[i] += W_out[i][j] * h_new[j];
}
softmax(logits, VOCAB_OUT);
int pred = 0;
for (i = 1; i < VOCAB_OUT; i++)
if (logits[i] > logits[pred]) pred = i;
printf(" t=%d input='%s' pred='%s' "
"target='%s' %s\n",
t,
t == 0 ? "<START>"
: out_vocab[target_tokens[t-1]],
out_vocab[pred],
out_vocab[target_tokens[t]],
pred == target_tokens[t]
? "OK" : "WRONG");
/* Teacher forcing: use target as next input */
dec_input = (float)target_tokens[t] / VOCAB_OUT;
for (i = 0; i < N_HID; i++) h[i] = h_new[i];
}
printf("\nWeights are random so predictions are "
"wrong.\n");
printf("With training, cross-entropy loss and "
"backprop through\n");
printf("both encoder and decoder, this learns "
"to reverse.\n");
return 0;
}
Figure 18-5 joins the two halves into one pipeline. The encoder section runs first and its hidden state accumulates steadily, moving from [−0.04, −0.03, −0.03, +0.03, +0.02, +0.00] after ‘a’ to [−0.10, −0.09, −0.12, +0.09, +0.06, −0.00] after ‘c’, which is printed underneath as the context vector. Those values are small because the weights are random and untrained, exactly as in every other untrained example in the book, and the sixth component sitting at −0.002 is small enough to be doing nothing at all.
The decoder then gets three of four wrong, and the manner of the failure is worth a glance before we move past it. It predicts ‘A’ at every single step, regardless of what it was fed, which makes it a constant function rather than a model. The one correct answer at t=2 is coincidence, since the target there happens to be ‘A’ as well. An untrained decoder with random weights and a decaying hidden state ends up with one output class marginally ahead of the others and simply emits it forever, which is the same degenerate behavior the untrained decoder showed when its probabilities flattened to a uniform 0.17.
What to look at instead is the input column. At t=0 the input is , at t=1 it is ‘C’, at t=2 it is ‘B’ and at t=3 it is ‘A’, and those are the target tokens rather than the predictions. Teacher forcing is running, so the decoder receives the correct history at every step no matter how badly it performs. Turn it off and the input column would read , ‘A’, ‘A’, ‘A’, because the model would be conditioning on its own constant output.
The two halves hold entirely separate weights, which the code makes explicit with enc_wx and dec_wx as different arrays. They meet at exactly one point, where the encoder’s final hidden state becomes the decoder’s initial hidden state, and that single connection is also the only path the gradient has. During training the loss is computed on the decoder outputs, flows back through every decoder step, arrives at the context vector, and continues back through every encoder step. The encoder never sees the loss directly and learns entirely through what the decoder manages to do with the vector it was handed.
18.7 The Bottleneck Problem
Everything so far has treated the context vector as adequate. This section tests that, and the test has to be chosen carefully, because measuring how a context vector looks at different lengths is not the same as measuring how much information it holds.
The question that matters is whether the first token still influences the context vector after the whole sequence has been read. If it does not, then early information has been lost regardless of what the vector looks like. So this program encodes a random sequence, encodes a second copy that differs only in the sign of the very first token, and measures how far apart the two context vectors end up.
/* 100_Bottleneck.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float my_tanh(float z)
{
if (z < -20) return -1;
float e = expf(-2 * z);
return (1 - e) / (1 + e);
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N_HID 4
#define MAX_LEN 50
/* Encode a sequence and return the final hidden
state */
static void encode(const float wx[N_HID],
const float wh[N_HID][N_HID],
const float bh[N_HID],
const float *x, int len,
float out[N_HID])
{
float h[N_HID] = {0}, hn[N_HID];
int t, i, j;
for (t = 0; t < len; t++) {
for (i = 0; i < N_HID; i++) {
float z = bh[i] + wx[i] * x[t];
for (j = 0; j < N_HID; j++)
z += wh[i][j] * h[j];
hn[i] = my_tanh(z);
}
for (i = 0; i < N_HID; i++) h[i] = hn[i];
}
for (i = 0; i < N_HID; i++) out[i] = h[i];
}
static float dist(const float a[N_HID],
const float b[N_HID])
{
float s = 0;
int i;
for (i = 0; i < N_HID; i++)
s += (a[i] - b[i]) * (a[i] - b[i]);
return sqrtf(s);
}
static float norm(const float a[N_HID])
{
float s = 0;
int i;
for (i = 0; i < N_HID; i++) s += a[i] * a[i];
return sqrtf(s);
}
int main(void)
{
float wx[N_HID], wh[N_HID][N_HID], bh[N_HID];
float seq[MAX_LEN], alt[MAX_LEN];
float ctx[N_HID], ctx_alt[N_HID];
int lengths[] = { 3, 5, 10, 20, 50 };
int n_lengths = 5;
int li, t, i, j;
srand(42);
for (i = 0; i < N_HID; i++) {
wx[i] = randf()*0.4f-0.2f;
bh[i] = 0;
for (j = 0; j < N_HID; j++)
wh[i][j] = randf()*0.4f-0.2f;
}
printf("Does the first token still affect the "
"context\n");
printf("vector after the whole sequence is "
"read?\n\n");
printf(" length |context| change from "
"relative\n");
printf(" at end flipping x0 "
"change\n");
printf(" ------ --------- ----------- "
"---------\n");
for (li = 0; li < n_lengths; li++) {
int len = lengths[li];
/* One random sequence, and a copy whose only
difference is the very first token */
srand(7);
for (t = 0; t < len; t++) {
seq[t] = randf() * 2.0f - 1.0f;
alt[t] = seq[t];
}
alt[0] = -seq[0];
encode(wx, wh, bh, seq, len, ctx);
encode(wx, wh, bh, alt, len, ctx_alt);
float n0 = norm(ctx);
float d = dist(ctx, ctx_alt);
printf(" %4d %.5f %.3e %.3e\n",
len, n0, d, n0 > 0 ? d / n0 : 0.0f);
}
printf("\nFlipping the sign of the first token "
"barely\n");
printf("moves the context once the sequence is "
"long.\n");
printf("The encoder has not saturated, since the "
"state\n");
printf("norm stays small throughout. The "
"information is\n");
printf("simply gone, overwritten by everything "
"after.\n\n");
printf("This is the bottleneck. The fix is to "
"stop\n");
printf("compressing everything into one vector "
"and\n");
printf("let the decoder look back "
"at ALL encoder\n");
printf("hidden states. That is attention, "
"Chapter 19.\n");
return 0;
}
Figure 18-6 measures how much the first token still affects the context vector. The relative change column is the answer and it collapses by nearly three orders of magnitude across the table. At length 3 flipping the sign of the first token moves the context vector by 1.097 of its own magnitude, which is to say the change is larger than the vector itself, and no wonder given that the first token is a third of the entire input. At length 5 the figure is 2.120e-01, so about a fifth. At length 10 it is 1.228e-03, roughly a thousandth, and by length 20 the two context vectors are bit for bit identical and the change is exactly zero. Twenty tokens in, the first token has no measurable influence whatsoever on what the encoder hands to the decoder. No amount of cleverness in the decoder recovers it, because it is not there.
Look at the context norm column while you are here, because it rules out the explanation people reach for first. The norms run 0.05971, 0.03431, 0.02183, 0.01302 and 0.10562, every one of them far below the range where tanh flattens out, which needs components approaching 1. This is not a vector that has been filled to capacity. It is a vector whose contents keep getting overwritten, because every new token passes the state through the recurrent weights again and each pass attenuates whatever was already there. The information is not compressed into the vector, it is discarded on the way through.
Notice too that the norm does not shrink monotonically, since length 50 comes out at 0.10562, the largest in the column. That is the giveaway. If the vector were filling up you would expect its magnitude to track the amount of input, and it does not, because the magnitude is set almost entirely by the last few tokens rather than by the whole sequence.
That distinction changes what a fix would look like. If the problem were saturation then a larger hidden state would solve it, and a larger state does buy some room. But the decay here is multiplicative per step, so doubling the hidden size buys a constant factor against an exponential loss, which is the same losing trade Chapter 14 described for gradients travelling backward.
The actual fix is to stop discarding the intermediate states. The encoder computes a hidden state at every input position and then throws all of them away except the last, which is a strange thing to do given what we have just measured. Keep them all, hand the whole set to the decoder, and let the decoder decide at each step which input positions matter for the token it is currently generating. That is the attention mechanism, and it is Chapter 19.
18.8 Key Takeaways
The encoder-decoder architecture separates reading from writing so that input and output lengths become independent. The encoder compresses the input into a fixed-size context vector and the decoder generates from it until it emits an end token.
The context vector is the only thing crossing between the halves. Its size does not grow with the input, which is what makes variable lengths possible and also what limits the architecture.
The decoder is autoregressive, feeding each generated token back as the next input. We watched an untrained decoder’s hidden state decay to near zero within four steps, after which it predicted a uniform 0.17 across all six tokens, and the joined pipeline showed another emitting the same token at every step.
Autoregressive decoding compounds errors, because a wrong token becomes the input that conditions every step after it. We measured the same flawed decoder scoring 1 of 5 autoregressively and 4 of 5 with teacher forcing, on an identical model.
Teacher forcing feeds the target token rather than the prediction during training. It buys stability and costs exposure bias, since the model trains on perfect inputs and runs on its own imperfect ones.
Encoder and decoder hold separate weights and meet at one point, where the final encoder state becomes the initial decoder state. That single connection is also the only route the gradient takes back into the encoder.
The bottleneck is real and we measured it. Flipping the first token of the input changed the context vector by 1.097 of its own magnitude at length 3, by 1.228e-03 at length 10, and by exactly zero at length 20.
The cause is overwriting rather than saturation. The context norms stayed between 0.013 and 0.106, nowhere near the range where tanh flattens, and the longest input produced the largest norm rather than the most saturated one.
The fix is to keep the encoder’s intermediate hidden states instead of discarding them, and let the decoder choose which ones matter at each output step. That is attention, built in Chapter 19.
Sutskever and colleagues published this architecture in 2014 and it carried neural machine translation for several years. Transformers have replaced it for most tasks, but the encoder-decoder split survives inside many of them.
18.9 Exercises
Implement the reverse task (input “abc”, output “cba”) with full training using BPTT through both encoder and decoder. Start with sequences of length 3.
What happens if you use a bidirectional encoder? The context vector would be the concatenation of the forward and backward final states. Does this help with the bottleneck?
Try using the context vector as input to every decoder step (concatenated with the token embedding) instead of only as the initial hidden state. Does this help?
Measure the reconstruction error by encoding a sequence, then decode it, and compare the output to the input. How does error grow with sequence length?
Implement beam search for decoding, where instead of greedily taking the best token at each step, keep track of the top-k sequences and expand each one. Does this improve output quality?
The bottleneck problem in 100_Bottleneck.c shows the context vectors converging. Increase N_HID to 16 and 64. Does a larger hidden state delay the saturation? What is the relationship between hidden size and maximum effective input length?