Dot-Product Attention
Similarity weighting, the core idea
19.1 What You Will Learn
Chapter 18 finished by measuring the bottleneck rather than describing it. Flipping the first token of a twenty token input changed the encoder’s context vector by exactly nothing, so whatever the decoder needed from the start of the sentence was already gone before decoding began. The encoder computes a hidden state at every input position and then discards all of them except the last, which is a strange thing to do once you have seen the measurement.
Attention keeps them. Instead of one context vector for the whole sequence, the decoder holds the entire set of encoder states and builds a fresh context at every output step by deciding which of them matter right now. Different output words draw on different input positions, so nothing has to be compressed into a fixed budget and no early information is overwritten.
The mechanism is smaller than its reputation suggests, and this chapter builds it in three pieces. Similarity by dot product comes first, then softmax to turn raw similarities into weights, then a weighted sum to produce the context. We then plug the result into the encoder-decoder from Chapter 18, and finish with the one practical correction that separates the textbook version from the version that actually trains. Everything from here to the transformer in Chapter 24 is a refinement of what this chapter builds.
19.2 The Idea
Suppose you are translating “the cat sat on the mat” into French. When the decoder is producing “chat” it should be looking at the encoder state for “cat”, and when it reaches “tapis” it should be looking at “mat”. Those are different positions in the input, needed at different moments, and a single context vector cannot be both at once.
So treat it as a lookup. At each decoder step the decoder holds a state describing what it is trying to produce, and the encoder holds one state per input position describing what is there. Compare the decoder state against every encoder state, score how well each one matches, and build the context out of the high scoring ones. The comparison happens fresh at every step, so the answer is allowed to change.
That leaves one question, which is how to measure a match between two vectors. The simplest answer available is the dot product, and it turns out to be good enough that the transformer still uses it a decade later.
19.3 Dot Product as Similarity
The dot product measures alignment. Two vectors pointing the same way produce a large positive number, perpendicular vectors produce zero, and vectors pointing in opposite directions produce a negative number. That is the whole of the similarity measure, and the program scores one query against five keys chosen to span the range.
/* 101_Dot_Product.c */
#include <stdio.h>
#include <math.h>
#define DIM 3
/* The label follows the score, so a reader can
check it rather than take it on trust */
static const char *label(float d)
{
if (d > 1.2f) return "same direction";
if (d > 0.8f) return "similar";
if (d > 0.2f) return "partly aligned";
if (d > -0.2f) return "orthogonal";
return "opposite";
}
static float dot(const float a[DIM], const float b[DIM])
{
float sum = 0;
int i;
for (i = 0; i < DIM; i++)
sum += a[i] * b[i];
return sum;
}
int main(void)
{
/* A query vector (what the decoder is looking
for) */
float query[DIM] = { 1.0f, 0.0f, 0.5f };
/* Five key vectors (encoder hidden states) */
float keys[5][DIM] = {
/* identical to query */
{ 1.0f, 0.0f, 0.5f },
{ 0.8f, 0.1f, 0.4f }, /* similar */
{ 0.0f, 1.0f, 0.0f }, /* orthogonal */
{ -0.5f, 0.2f, -0.3f }, /* opposite-ish */
{ 0.4f, 0.0f, 0.6f }, /* partly aligned */
};
int i;
printf("Dot product as similarity:\n\n");
printf(" Query: [%.1f, %.1f, %.1f]\n\n",
query[0], query[1], query[2]);
printf(" Key Dot product "
"Interpretation\n");
printf(" ------------------- ----------- "
"---------------\n");
for (i = 0; i < 5; i++) {
float d = dot(query, keys[i]);
printf(" [%+4.1f, %+4.1f, %+4.1f] %+7.2f "
" %s\n",
keys[i][0], keys[i][1], keys[i][2],
d, label(d));
}
printf("\nA higher dot product means more similar "
"and more relevant.\n");
printf("The decoder pays more attention to "
"similar states.\n");
return 0;
}
Figure 19-1 scores one query against five keys. Read the scores against the vectors that produced them. The query is [1.0, 0.0, 0.5], and the first key is identical to it, scoring +1.25, which is simply the query dotted with itself. The third key is [0.0, 1.0, 0.0], perpendicular to the query in every component, and scores exactly +0.00. The fourth points broadly the other way and scores −0.65. Between those, the second key scores +1.00 and the fifth scores +0.70, which places them where their geometry says they belong.
The interpretation column is derived from the score rather than attached to each row by hand, which matters more than it sounds. A hardcoded label tells you what the author thought about the vector, while a computed one tells you what the number actually says, and a reader can check the second kind. Change any key in the array and the label changes with it.
Notice what the dot product does not do. It is sensitive to length as well as direction, so a key pointing exactly along the query but twice as long would score twice as high without being any more relevant. Exercise 3 asks you to replace it with cosine similarity, which normalizes that away. Attention keeps the unnormalized version anyway, partly because it is cheaper and partly because the network can learn to control the magnitudes itself, and the end of the chapter deals with what happens when it does not.
19.4 Softmax Weights
Raw dot products are unusable as weights. They can be negative, as the fourth key just was, and they do not sum to anything in particular, so multiplying encoder states by them would produce a context vector of arbitrary scale pointing partly backwards. Softmax fixes both, turning any set of real numbers into positive values that sum to 1.
/* 102_Attention_Weights.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#define DIM 3
#define N_KEYS 5
static float dot(const float a[DIM], const float b[DIM])
{
float sum = 0;
int i;
for (i = 0; i < DIM; i++) sum += a[i] * b[i];
return sum;
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
int main(void)
{
float query[DIM] = { 1.0f, 0.0f, 0.5f };
float keys[N_KEYS][DIM] = {
{ 1.0f, 0.0f, 0.5f },
{ 0.8f, 0.1f, 0.4f },
{ 0.0f, 1.0f, 0.0f },
{ -0.5f, 0.2f, -0.3f },
{ 0.6f, 0.0f, 0.8f },
};
/* Compute raw scores */
float scores[N_KEYS];
int i;
for (i = 0; i < N_KEYS; i++)
scores[i] = dot(query, keys[i]);
printf("Raw dot-product scores:\n [");
for (i = 0; i < N_KEYS; i++)
printf("%.2f%s", scores[i],
i < N_KEYS-1 ? ", " : "");
printf("]\n\n");
/* Convert to weights via softmax */
softmax(scores, N_KEYS);
printf("After softmax (attention weights):\n [");
for (i = 0; i < N_KEYS; i++)
printf("%.3f%s", scores[i],
i < N_KEYS-1 ? ", " : "");
printf("]\n\n");
float sum = 0;
for (i = 0; i < N_KEYS; i++) sum += scores[i];
printf("Sum of weights: %.4f\n", sum);
printf("\nThe most similar key gets the highest "
"weight.\n");
printf("All weights are positive and sum to 1.\n");
return 0;
}
Figure 19-2 turns those raw scores into weights that sum to one. The five raw scores [1.25, 1.00, 0.00, −0.65, 1.00] come out as weights [0.334, 0.260, 0.096, 0.050, 0.260], summing to 1.0000 as the program confirms. Every weight is positive, including the one that came from a negative score, because exponentiating makes everything positive before the division normalizes.
Look at how much the ranking is preserved and how much it is compressed. The best score was 1.25 and the worst was −0.65, a spread of 1.9, but the weights run from 0.334 down to 0.050, so the winner takes only about seven times the loser’s share rather than dominating outright. The two keys that tied at 1.00 receive identical weights of 0.260 each, which is the behavior you want and worth confirming rather than assuming.
That softness is the point. A hard argmax would pick one encoder state and discard the other four, which is not differentiable and gives the network nothing to learn from. Softmax lets the decoder read mostly from one position while still receiving a trickle from the others, and the gradient reaches every key. We show later what goes wrong when the scores grow large enough that softmax stops being soft.
19.5 The Weighted Sum
Weights on their own do nothing. The output of attention is the encoder states themselves, blended in proportion to those weights, which gives a single vector of the same width as one encoder state but containing a mixture drawn from all of them.
/* 103_Weighted_Sum.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#define DIM 4
#define N_KEYS 4
static float dot(const float *a, const float *b, int n)
{
float sum = 0;
int i;
for (i = 0; i < n; i++) sum += a[i] * b[i];
return sum;
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
int main(void)
{
/* Decoder state (the query) */
float query[DIM] = { 0.5f, 0.8f, -0.2f, 0.3f };
/* Encoder hidden states at each position */
float keys[N_KEYS][DIM] = {
{ 0.1f, 0.9f, -0.1f, 0.2f }, /* "the" */
{ 0.7f, 0.2f, 0.5f, 0.8f }, /* "cat" */
{ 0.3f, 0.1f, -0.4f, 0.1f }, /* "sat" */
{ 0.0f, 0.5f, 0.2f, -0.1f }, /* "on" */
};
/* Values (same as keys in basic attention) */
float values[N_KEYS][DIM];
int i, j;
for (i = 0; i < N_KEYS; i++)
for (j = 0; j < DIM; j++)
values[i][j] = keys[i][j];
const char *words[] = { "the", "cat", "sat", "on" };
/* Step 1: compute scores */
float scores[N_KEYS];
for (i = 0; i < N_KEYS; i++)
scores[i] = dot(query, keys[i], DIM);
/* Step 2: softmax */
softmax(scores, N_KEYS);
/* Step 3: weighted sum of values */
float context[DIM] = { 0 };
for (i = 0; i < N_KEYS; i++)
for (j = 0; j < DIM; j++)
context[j] += scores[i] * values[i][j];
printf("Attention mechanism step by step:\n\n");
printf("1. Compute similarity (dot product):\n");
for (i = 0; i < N_KEYS; i++)
printf(" query . key[\"%s\"]"
" = %.3f\n", words[i],
dot(query, keys[i], DIM));
printf("\n2. Apply softmax (get weights):\n");
for (i = 0; i < N_KEYS; i++)
printf(" weight[\"%s\"] = %.3f\n",
words[i], scores[i]);
printf("\n3. Weighted sum of values, the "
"context vector\n");
printf(" context = ");
for (i = 0; i < N_KEYS; i++)
printf("%.3f*v[\"%s\"] %s", scores[i], words[i],
i < N_KEYS-1 ? "+ " : "");
printf("\n context = [%.3f, %.3f, %.3f, %.3f]\n",
context[0], context[1],
context[2], context[3]);
printf("\nThe context vector is a blend of all "
"encoder states,\n");
printf("weighted by relevance to the current "
"decoder step.\n");
printf("It replaces the single bottleneck vector "
"from Chapter 18.\n");
return 0;
}
Figure 19-3 has all three stages, score, normalize and blend. The program prints them for one decoder step over a four word input. The similarity scores come out at 0.850, 0.650, 0.340 and 0.330, softmax turns them into 0.332, 0.272, 0.199 and 0.197, and the weighted sum produces [0.283, 0.472, 0.062, 0.284].
Read the weights before moving on, because they are flatter than the phrase attention suggests. The top weight is 0.332 and the bottom is 0.197, so the most relevant word contributes only about a third and the least relevant still contributes a fifth. This is not a spotlight picking out one word, it is a mild preference. With scores this close together, softmax has very little to work with, and the context vector is close to an average of all four encoder states.
That is the normal case for an untrained or weakly discriminating model, and seeing it before the sharper examples arrive is useful. Training is what pulls the scores apart, and the next section uses hand chosen states to show what the mechanism looks like once they are.
Three lines of arithmetic, then. Score, normalize, blend. Everything in the remainder of this book that is called attention, including multi-head attention and self-attention and the whole transformer, is this same pattern with different things plugged into the three slots.
19.6 Attention in a Seq2Seq Decoder
Now plug it into the encoder-decoder. The input is “I love big fat cats” and the decoder produces three French words, and at every step it attends over all five encoder states rather than reading a single compressed vector.
The encoder states and decoder queries here are chosen rather than trained, in the same spirit as the hand picked weights throughout Chapters 15 and 16. A randomly initialized model produces the flat, near uniform weights we just saw in 103_Weighted_Sum.c, which demonstrates that the arithmetic runs but hides what the mechanism is for.
/* 104_Seq2seq_Attention.c */
#include <stdio.h>
#include <math.h>
#define ENC_LEN 5
#define DEC_LEN 3
#define DIM 4
static void softmax(float *x, int n)
{
float mx = x[0], s = 0;
int i;
for (i = 1; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
static float dot(const float *a, const float *b, int n)
{
float s = 0;
int i;
for (i = 0; i < n; i++) s += a[i] * b[i];
return s;
}
int main(void)
{
const char *in_words[ENC_LEN] = {
"I", "love", "big", "fat", "cats"
};
const char *out_words[DEC_LEN] = {
"j'aime", "gros", "chats"
};
/* Encoder states, one per input word. Chosen rather
than trained,
so the alignment is easy to read. */
float enc[ENC_LEN][DIM] = {
{ 0.9f, 0.1f, 0.0f, 0.1f }, /* I */
{ 0.8f, 0.3f, 0.1f, 0.0f }, /* love */
{ 0.1f, 0.9f, 0.2f, 0.0f }, /* big */
{ 0.0f, 0.8f, 0.3f, 0.1f }, /* fat */
{ 0.0f, 0.1f, 0.2f, 0.9f }, /* cats */
};
/* One decoder query per output word */
float dec[DEC_LEN][DIM] = {
{ 2.5f, 0.0f, 0.0f, 0.0f }, /* j'aime */
{ 0.0f, 2.5f, 0.0f, 0.0f }, /* gros */
{ 0.0f, 0.0f, 0.0f, 2.5f }, /* chats */
};
float w[ENC_LEN], context[DIM];
int t, i, j, focus;
printf("Decoder with attention, "
"%d output steps\n\n",
DEC_LEN);
for (t = 0; t < DEC_LEN; t++) {
for (i = 0; i < ENC_LEN; i++)
w[i] = dot(dec[t], enc[i], DIM);
softmax(w, ENC_LEN);
for (j = 0; j < DIM; j++) {
context[j] = 0;
for (i = 0; i < ENC_LEN; i++)
context[j] += w[i] * enc[i][j];
}
focus = 0;
for (i = 1; i < ENC_LEN; i++)
if (w[i] > w[focus]) focus = i;
printf(" Step %d, generating \"%s\"\n",
t, out_words[t]);
printf(" weights: ");
for (i = 0; i < ENC_LEN; i++)
printf("%s=%.2f ", in_words[i], w[i]);
printf("\n focus: \"%s\" at %.0f%%\n",
in_words[focus], w[focus] * 100.0f);
printf(" context: [%+.3f, %+.3f, %+.3f, "
"%+.3f]"
"\n\n",
context[0], context[1], context[2],
context[3]);
}
printf("Each step produces its "
"own distribution,\n");
printf("so a different context "
"vector reaches the\n");
printf("output layer every time. Chapter 18 had "
"one\n");
printf("context vector for the whole sequence.\n");
return 0;
}
Figure 19-4 has three decoder steps and three different attention distributions. They are visibly different, which is the entire claim of this chapter. Generating “j’aime” puts 0.47 on “I” and 0.37 on “love”, together holding 84 percent of the weight, with the remaining three words splitting the rest. Generating “gros” shifts to 0.44 on “big” and 0.34 on “fat”, and “I” has collapsed from 0.47 to 0.06. Generating “chats” puts 0.68 on “cats”, the sharpest single focus in the table.
The context vectors underneath change accordingly. Step 0 produces [+0.723, +0.259, +0.074, +0.097], dominated by its first component, while step 1 produces [+0.176, +0.712, +0.213, +0.094] with the mass moved into the second, and step 2 produces [+0.146, +0.235, +0.184, +0.626] with it moved into the fourth. Three different vectors reach the output layer from the same input sentence.
Set that against Chapter 18, where the decoder received one context vector and used the same one at every step. There the input was compressed once and the compression was lossy in a way the last chapter measured directly. Here nothing is compressed at all, since every encoder state is still available in full, and the decoder simply decides how much of each to read. The bottleneck has not been widened, it has been removed.
Notice that the weights never reach 0 or 1. Even at step 2, where “cats” takes 0.68, the other four words keep between 0.07 and 0.09 apiece. Attention is soft by construction, and the model is always reading a little from everywhere, which is what keeps it differentiable.
19.7 Scaled Dot-Product
One practical problem remains and it only appears at realistic dimensions. A dot product sums d products, so its typical magnitude grows with d, and by the time d reaches 512 the scores are large enough to push softmax into a corner where one weight is nearly 1 and the rest are nearly 0.
That is bad for two reasons. The context vector becomes a copy of a single encoder state, which throws away the blending the mechanism exists to provide, and the gradient of softmax in that regime is almost zero, so no key receives a useful learning signal. The fix is a single division.
The square root is the right correction because of how the sum behaves. A dot product over d dimensions adds up d independent products, and a sum of d independent terms grows in typical magnitude like the square root of d rather than like d itself. Dividing by that same factor leaves the scores at roughly the same scale whatever the dimension, so a model with d of 512 sees a softmax input no larger than a model with d of 8. Dividing by d instead would overcorrect and flatten the scores toward each other as the dimension grew.
/* 105_Scaled.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#define N_KEYS 4
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
static void show_weights(const char *label,
const float *scores, int n)
{
float w[N_KEYS];
int i;
for (i = 0; i < n; i++) w[i] = scores[i];
softmax(w, n);
printf(" %-12s scores=[%+5.1f, %+5.1f, "
"%+5.1f, %+5.1f] "
"weights=[%.3f, %.3f, %.3f, %.3f]\n",
label, scores[0], scores[1],
scores[2], scores[3],
w[0], w[1], w[2], w[3]);
}
int main(void)
{
printf("Effect of scaling on "
"attention weights:\n\n");
/* Small dimension (d=4): scores are small */
float small_scores[N_KEYS] =
{ 1.2f, 0.8f, 0.3f, -0.5f };
show_weights("d=4", small_scores, N_KEYS);
/* Large dimension (d=512): scores are much
larger */
float large_scores[N_KEYS] =
{ 15.0f, 10.0f, 3.0f, -6.0f };
show_weights("d=512 (raw)", large_scores, N_KEYS);
/* After scaling by 1/sqrt(512) = 1/22.6 */
float scale = 1.0f / sqrtf(512.0f);
float scaled[N_KEYS];
int i;
for (i = 0; i < N_KEYS; i++)
scaled[i] = large_scores[i] * scale;
show_weights("d=512 (scaled)", scaled, N_KEYS);
printf("\nWithout scaling, large-d scores push "
"softmax into\n");
printf("near-one-hot mode. Only one key gets "
"all the weight.\n");
printf("Scaling by 1/sqrt(d) keeps the "
"distribution soft,\n");
printf("allowing gradients to flow to all keys.\n");
printf("\nThis is the 'scaled' in 'scaled "
"dot-product attention'.\n");
printf("Every transformer uses it.\n");
return 0;
}
Figure 19-5 shows why large dimensions need the correction. The three rows show the same shape of scores at two dimensions. At d=4 the scores are [+1.2, +0.8, +0.3, −0.5] and the weights come out [0.443, 0.297, 0.180, 0.081], which is a reasonable spread with a clear favorite and nothing shut out.
The middle row is the same relative pattern at d=512 without scaling, where the scores are [+15.0, +10.0, +3.0, −6.0] and the weights collapse to [0.993, 0.007, 0.000, 0.000]. The top key has taken 99.3 percent and the bottom two round to zero at three decimal places. That is a hard selection wearing the costume of a soft one, and the two keys reading 0.000 will receive essentially no gradient.
The last row divides those same scores by sqrt(512), which is about 22.6, giving [+0.7, +0.4, +0.1, −0.3] and weights of [0.359, 0.288, 0.211, 0.142]. The ranking is untouched, since dividing every score by the same positive number cannot reorder them, but the distribution is soft again and every key is back in play. Compare that bottom row against the d=4 row at the top and they are in the same territory, which is exactly what the scaling is for. It makes behavior at large d resemble behavior at small d.
The choice of sqrt(d) rather than d or some other function is not arbitrary. If the query and key components are roughly independent with unit variance, their dot product over d terms has a variance of d and therefore a standard deviation of sqrt(d), so dividing by sqrt(d) returns the scores to roughly unit scale regardless of dimension.
19.8 The Complete Formula
Everything built in this chapter compresses into one line.
Figure 19-6 is that line drawn as a pipeline. What the picture adds to the formula is the path V takes, since V skips the scoring entirely and joins only at the final multiply. Q and K decide how much weight each position gets, and V is untouched until those weights are applied to it.
Q is the query, K is the keys, V is the values, and d is the dimension being summed over. Reading it right to left recovers the three steps. Q times K transposed produces one similarity score per key, dividing by sqrt(d) applies the correction just described, softmax turns the scores into weights, and multiplying by V takes the weighted sum.
This is the formula from Vaswani and colleagues in 2017, and every transformer in existence contains it. What changes between applications is not the formula but where the three inputs come from. In the seq2seq attention we just built, Q is the decoder state and K and V are both the encoder states, which is why the program used the same array for both. In the self-attention of Chapter 20, all three come from the same sequence, so a position attends to other positions of the input it belongs to. In the cross-attention inside an encoder-decoder transformer, Q comes from the decoder and K and V from the encoder, exactly as here.
The separation of K from V is worth noticing even though this chapter kept them equal. Keys are what you match against and values are what you retrieve, and there is no requirement that they be the same thing. A model can learn to match on one aspect of a position while returning a different aspect of it, and Chapter 21 puts that freedom to use.
19.9 Key Takeaways
Attention replaces the fixed context vector with a lookup performed fresh at every decoder step. Nothing is compressed, because all encoder states remain available and the decoder chooses how much of each to read.
The mechanism is three operations. Score every key against the query with a dot product, normalize the scores with softmax, and take the weighted sum of the values.
The dot product measures alignment and is sensitive to length as well as direction. We scored an identical key at +1.25, a perpendicular one at exactly +0.00, and an opposing one at −0.65.
Softmax makes the weights positive and summing to 1 while compressing the spread. Raw scores from 1.25 down to −0.65 became weights from 0.334 down to 0.050, and two keys that tied at 1.00 received identical weights of 0.260.
Weights are never 0 or 1 in practice, which is what keeps attention differentiable. Even the sharpest decoder step left between 0.07 and 0.09 on every unselected word.
Different decoder steps produce different distributions. We measured 0.47 on “I” while generating one word and 0.06 on the same word two steps later, with the context vector shifting its mass accordingly.
Large dimensions push softmax into near one-hot behavior. At d=512 the unscaled weights were [0.993, 0.007, 0.000, 0.000], which is a hard selection and gives almost no gradient to the losing keys.
Dividing by sqrt(d) restores a usable spread without changing the ranking, since scaling every score by the same positive number cannot reorder them. The same scores rescaled gave [0.359, 0.288, 0.211, 0.142].
The complete formula is Attention(Q, K, V) = softmax(Q * K^T / sqrt(d)) * V. Only the source of Q, K and V changes between seq2seq attention, self-attention and cross-attention.
Keys and values are equal in this chapter but need not be. Keys are what a query matches against, values are what gets retrieved, and separating them lets a model match on one property and return another.
19.10 Exercises
Compute dot-product attention by hand for Q = [1, 0], K = [[1, 0], [0, 1], [−1, 0]], V = [[10, 0], [0, 10], [−10, 0]]. What is the context vector? Why?
What happens to the attention weights when all keys are identical? What does the context vector become?
Replace dot-product similarity with cosine similarity (dot product divided by the product of norms). How does this change the attention weights?
Add attention to the full seq2seq model from Chapter 18, which is 099_Seq2seq.c. Concatenate the attention context with the decoder hidden state before the output layer.
Compute the gradient of the context vector with respect to the query. How does the gradient flow through the softmax?
Plot the attention weights across all decoder steps for a 10-word input. You should see a diagonal-like pattern where each decoder step focuses on a different input position (for a reversal or translation task).