Positional Encoding
Sinusoidal, learned, and RoPE
22.1 What You Will Learn
Attention as built over the last three chapters has a hole in it, and the hole is order. Every query is compared against every key by a dot product, and a dot product does not care where either vector came from. Shuffle the input positions and the attention weights follow the vectors around without anything in the mechanism noticing that a sequence has been rearranged. For language that is fatal, since “dog bites man” and “man bites dog” contain the same three words and mean opposite things.
Positional encoding closes the hole by putting position information into the representation, and there are three ways to do it that all remain in use. The original transformer computed a fixed pattern of sines and cosines and added it to the embeddings. GPT-2 and BERT dropped the formula and simply learned a vector per position. Modern models rotate the queries and keys by an angle that depends on position, which turns out to encode relative distance rather than absolute location.
This chapter builds all three and measures what separates them. Section 22.3 demonstrates the order blindness rather than asserting it, 22.4 through 22.6 cover the two additive approaches, 22.7 builds RoPE and confirms its relative position property numerically, and 22.8 sets the three side by side.
22.2 Why Attention Has No Position
The recurrent networks of Chapters 13 through 17 got position for free. A hidden state at step 5 has been through five updates, and there is no way for it to be anything other than the state at step 5, because the updates happened in order and each depended on the last.
Attention throws that away deliberately. Every position is computed at the same time from the same matrices, so the query at position 0 and the query at position 5 go through identical arithmetic. That parallelism is the whole reason transformers train faster than recurrent networks, and the price is that the mechanism has no idea which position is which. Swap two input vectors and their outputs swap with them, leaving everything else untouched. The technical term is permutation equivariance, and for a set of items it is exactly the property you want. For a sentence it is a disaster.
So position has to be added from outside, either mixed into the input before attention sees it or applied to the queries and keys during the comparison. Both routes appear in this chapter.
Figure 22-1 marks the spot. Everything above the highlighted band is attention, feed-forward layers and normalisation, repeated N times on each side, and none of it has any notion of order. Position enters at one place only, added to the embeddings before the first layer sees them, on the encoder and decoder sides alike.
That the addition happens once and at the bottom is what makes the rest of the chapter matter. Get the encoding wrong and every layer above inherits the mistake, since nothing further up ever recovers position independently.
22.3 The Problem Without Position
Rather than argue for order blindness, run the same three word sentence twice with the first and third words swapped, and compare the attention weights.
/* 115_No_Position.c */
#include <stdio.h>
#include <math.h>
#include <float.h>
#define DIM 3
#define SEQ 3
static float dot(const float *a, const float *b, int n)
{
float s = 0;
int i;
for (i = 0; i < n; i++) s += a[i] * b[i];
return s;
}
static void softmax(float *x, int n)
{
float mx = -FLT_MAX, s = 0;
int i;
for (i = 0; i < n; i++) if (x[i] > mx) mx = x[i];
for (i = 0; i < n; i++) {
x[i] = expf(x[i] - mx);
s += x[i];
}
for (i = 0; i < n; i++) x[i] /= s;
}
static void run_attention(const char *label,
const float X[SEQ][DIM],
const char *names[SEQ])
{
float scale = 1.0f / sqrtf((float)DIM);
int i, j;
printf(" %s:\n", label);
for (i = 0; i < SEQ; i++) {
float scores[SEQ];
for (j = 0; j < SEQ; j++)
scores[j] = dot(X[i], X[j], DIM) * scale;
softmax(scores, SEQ);
printf(" %-5s attends to: ", names[i]);
for (j = 0; j < SEQ; j++)
printf("%s=%.2f ", names[j], scores[j]);
printf("\n");
}
}
int main(void)
{
float A[SEQ][DIM] = {
{ 1.0f, 0.2f, 0.5f }, /* dog */
{ 0.3f, 0.8f, 0.1f }, /* bites */
{ 0.5f, 0.4f, 0.9f }, /* man */
};
float B[SEQ][DIM] = {
/* man (was position 2) */
{ 0.5f, 0.4f, 0.9f },
/* bites (same position) */
{ 0.3f, 0.8f, 0.1f },
/* dog (was position 0) */
{ 1.0f, 0.2f, 0.5f },
};
const char *names_a[] = { "dog", "bites", "man" };
const char *names_b[] = { "man", "bites", "dog" };
printf("Attention without "
"positional encoding:\n\n");
run_attention("\"dog bites man\"", A, names_a);
printf("\n");
run_attention("\"man bites dog\"", B, names_b);
printf("\n Every row is IDENTICAL "
"across the two\n");
printf(" orderings. The model cannot tell "
"subject\n");
printf(" from object. It needs position.\n");
return 0;
}
Read the two blocks against each other and every row survives the swap unchanged. In “dog bites man”, dog attends to itself at 0.40, to bites at 0.26 and to man at 0.34. In “man bites dog”, dog attends to itself at 0.40, to bites at 0.26 and to man at 0.34. The numbers have moved across the display because the columns are printed in sentence order, but the relationship between any two words is identical.
The row for bites is worth reading separately because it shows the failure at its sharpest. It gives 0.32 to dog and 0.32 to man in both sentences, meaning the verb weights its subject and its object exactly the same. Nothing available to the model distinguishes the thing doing the biting from the thing being bitten.
This is not a weakness of these particular embeddings or these particular weights. It is arithmetic. Attention computes a function of the set of input vectors, and a set has no order, so no choice of weights can make the output depend on an ordering the input never carried. Fixing it means changing the input rather than the mechanism, which is what the rest of the chapter does.
22.4 Sinusoidal Positional Encoding
The original transformer generates a vector per position from a formula, using sine on the even dimensions and cosine on the odd ones, with the frequency falling as the dimension index rises.
The exponent is what spreads the frequencies out. At dimension 0 the divisor is 1, so the value completes a cycle every 2 pi positions. At the far end of a 512 dimensional model the divisor approaches 10000, so that dimension takes tens of thousands of positions to come round once.
/* 116_Sinusoidal.c */
#include <stdio.h>
#include <math.h>
#define D_MODEL 8
#define MAX_POS 10
static void sinusoidal_pe(int pos,
int d_model, float *pe)
{
int i;
for (i = 0; i < d_model; i++) {
float e = (float)(i / 2 * 2) / d_model;
float angle = pos / powf(10000.0f, e);
if (i % 2 == 0)
pe[i] = sinf(angle);
else
pe[i] = cosf(angle);
}
}
int main(void)
{
float pe[D_MODEL];
int pos, i;
printf("Sinusoidal positional encoding, "
"d_model=%d\n\n", D_MODEL);
printf(" pos ");
for (i = 0; i < D_MODEL; i++) printf(" dim%d ", i);
printf("\n --- ");
for (i = 0; i < D_MODEL; i++) printf("------ ");
printf("\n");
for (pos = 0; pos < 6; pos++) {
sinusoidal_pe(pos, D_MODEL, pe);
printf(" %2d ", pos);
for (i = 0; i < D_MODEL; i++)
printf("%+.3f ", pe[i]);
printf("\n");
}
/* Show that nearby positions have similar
encodings */
printf("\nDistance between position encodings:\n");
for (pos = 0; pos < 5; pos++) {
float pe_a[D_MODEL], pe_b[D_MODEL];
sinusoidal_pe(pos, D_MODEL, pe_a);
sinusoidal_pe(pos + 1, D_MODEL, pe_b);
float dist = 0;
for (i = 0; i < D_MODEL; i++) {
float d = pe_a[i] - pe_b[i];
dist += d * d;
}
dist = sqrtf(dist);
printf(" dist(%d, %d) = %.4f\n", pos,
pos + 1, dist);
}
printf("\nNearby positions have similar "
"encodings.\n");
printf("The distance stays constant rather than "
"growing.\n");
return 0;
}
The table shows the frequency spread directly. Dimension 0 runs +0.000, +0.841, +0.909, +0.141, −0.757, −0.959 across six positions, which is most of a full cycle in six steps. Dimension 2 covers only +0.000 to +0.479 over the same span, and dimension 6 has crawled from +0.000 to +0.005. The slow dimensions are almost constant here, which is not a fault but the design working, since they exist to distinguish position 50 from position 5000 rather than position 4 from position 5.
The distance measurements underneath are the more interesting result. The gap between consecutive positions comes out at 0.9641 for every pair tested, from dist(0,1) through dist(4,5). It does not grow, it does not shrink, and it does not depend on where in the sequence you measure. A model can therefore learn that a particular displacement in encoding space means one step, and that reading stays valid at position 5 and position 500.
That constancy is a consequence of the trigonometry rather than a coincidence. Because sin(a+b) and cos(a+b) expand into linear combinations of sin(a), cos(a), sin(b) and cos(b), the encoding at position p+k is a fixed linear transformation of the encoding at p, with the transformation depending only on k. A model can learn to attend three positions to the left as a single operation, rather than learning it separately for every starting point.
22.5 Adding Position to Embeddings
The positional vector is added to the token embedding rather than concatenated onto it, which surprises people the first time they see it. Concatenation would keep the two signals in separate dimensions and never let them interfere, at the cost of doubling the width of everything downstream.
/* 117_Add_Position.c */
#include <stdio.h>
#include <math.h>
#define D_MODEL 6
#define SEQ_LEN 4
static void sinusoidal_pe(int pos,
int d_model, float *pe)
{
int i;
for (i = 0; i < d_model; i++) {
float e = (float)(i / 2 * 2) / d_model;
float angle = pos / powf(10000.0f, e);
pe[i] = (i % 2 == 0)
? sinf(angle)
: cosf(angle);
}
}
int main(void)
{
/* Token embeddings, as if from the table */
float embeddings[SEQ_LEN][D_MODEL] = {
/* the */
{ 0.5f, -0.2f, 0.8f, 0.1f, -0.3f, 0.6f },
/* cat */
{ 0.3f, 0.7f, -0.1f, 0.4f, 0.5f, -0.2f },
/* sat */
{ -0.4f, 0.3f, 0.6f, -0.5f, 0.2f, 0.1f },
/* down */
{ 0.2f, -0.1f, 0.4f, 0.7f, -0.6f, 0.3f },
};
const char *words[] = { "the", "cat",
"sat", "down" };
int pos, i;
printf("Token embeddings + "
"positional encoding:\n\n");
printf(" Token embeddings (no position):\n");
for (pos = 0; pos < SEQ_LEN; pos++) {
printf(" %-5s [", words[pos]);
for (i = 0; i < D_MODEL; i++)
printf("%+.2f%s", embeddings[pos][i],
i < D_MODEL-1 ? "," : "");
printf("]\n");
}
printf("\n Positional encodings:\n");
for (pos = 0; pos < SEQ_LEN; pos++) {
float pe[D_MODEL];
sinusoidal_pe(pos, D_MODEL, pe);
printf(" pos %d [", pos);
for (i = 0; i < D_MODEL; i++)
printf("%+.2f%s", pe[i],
i < D_MODEL-1 ? "," : "");
printf("]\n");
}
printf("\n After addition (embedding + "
"position):\n");
for (pos = 0; pos < SEQ_LEN; pos++) {
float pe[D_MODEL];
sinusoidal_pe(pos, D_MODEL, pe);
printf(" %-5s [", words[pos]);
for (i = 0; i < D_MODEL; i++)
printf("%+.2f%s",
embeddings[pos][i] + pe[i],
i < D_MODEL-1 ? "," : "");
printf("]\n");
}
printf("\n The same word at two positions now "
"has\n");
printf(" different vectors. 'cat' at 1 differs\n");
printf(" from 'cat' at position 5.\n");
return 0;
}
Three blocks of numbers, and the third is the sum of the first two. The token “the” starts as [+0.50, −0.20, +0.80, +0.10, −0.30, +0.60], the encoding for position 0 is [+0.00, +1.00, +0.00, +1.00, +0.00, +1.00], and the result is [+0.50, +0.80, +0.80, +1.10, −0.30, +1.60]. Every odd dimension gained roughly 1.0, because cosine of zero is 1 and position 0 sits at angle zero in every frequency.
The consequence is the point of the section. The vector reaching attention for “cat” at position 1 is [+1.14, +1.24, −0.05, +1.40, +0.50, +0.80], and the same word at a different position would produce something different, so attention can now tell the two apart. Order has entered the representation without any change to the attention code.
Addition looks lossy and mostly is not, for a reason worth understanding. The embedding dimensions are learned, so the network is free to reserve capacity for the positional signal wherever it needs to, and the positional pattern is fixed and known so the network can learn to subtract it out where it is unwanted. In practice the two signals coexist, with lower layers leaning on position for local syntax and upper layers leaning on content.
Notice also that the encoding here is bounded between −1 and +1 while the embeddings are of similar scale, which is deliberate. An encoding an order of magnitude larger would swamp the content, and one an order of magnitude smaller would be ignored.
22.6 Learned Positional Embeddings
The alternative is to stop computing and start looking up. Treat position as a category, allocate a vector per position exactly as Chapter 10 allocated a vector per token, and let training decide what each one should contain.
That buys flexibility and costs a hard boundary, which this program demonstrates by asking for positions on both sides of it.
/* 118_Learned.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
#define D_MODEL 4
#define MAX_POS 8
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
/* Learned table, one row per position. Untrained
here, so the values are only placeholders. */
static float table[MAX_POS][D_MODEL];
/* Returns 0 and leaves out untouched when the
position is past the end of the table */
static int learned_pe(int pos, float out[D_MODEL])
{
int i;
if (pos < 0 || pos >= MAX_POS) return 0;
for (i = 0; i < D_MODEL; i++)
out[i] = table[pos][i];
return 1;
}
/* Sinusoidal needs no table, so it is defined
for every position there is */
static void sinusoidal_pe(int pos, float out[D_MODEL])
{
int i;
for (i = 0; i < D_MODEL; i++) {
float e = (float)(i / 2 * 2) / D_MODEL;
float angle = pos / powf(10000.0f, e);
out[i] = (i % 2 == 0)
? sinf(angle)
: cosf(angle);
}
}
int main(void)
{
float v[D_MODEL];
int pos, i;
srand(42);
for (pos = 0; pos < MAX_POS; pos++)
for (i = 0; i < D_MODEL; i++)
table[pos][i] = randf() * 0.4f - 0.2f;
printf("Learned table, d_model=%d, max_pos=%d\n\n",
D_MODEL, MAX_POS);
for (pos = 0; pos < MAX_POS; pos++) {
learned_pe(pos, v);
printf(" pos %d: [%+.3f, %+.3f, %+.3f, "
"%+.3f]\n",
pos, v[0], v[1], v[2], v[3]);
}
printf("\nAsking for positions past the table\n\n");
printf(" pos learned sinusoidal\n");
printf(" --- -------------- ----------\n");
for (pos = 6; pos <= 10; pos++) {
float s[D_MODEL];
int ok = learned_pe(pos, v);
sinusoidal_pe(pos, s);
printf(" %2d ", pos);
if (ok) printf("%+.3f %+.3f ", v[0], v[1]);
else printf("no entry ");
printf("%+.3f %+.3f\n", s[0], s[1]);
}
printf("\nThe table stops at %d. "
"Sinusoidal keeps\n",
MAX_POS - 1);
printf("going because it is a "
"formula rather than\n");
printf("a lookup, defined at every integer.\n\n");
printf("Table cost: %d x %d = %d floats\n",
MAX_POS, D_MODEL, MAX_POS * D_MODEL);
printf("At max_pos=2048, d_model=768 "
"that becomes\n");
printf("%d floats, or %d KB at 4 bytes each.\n",
2048 * 768, 2048 * 768 * 4 / 1024);
return 0;
}
The first block is the table, eight positions of four dimensions, untrained here so the values carry no meaning. The second block is where the argument lives. At positions 6 and 7 both approaches return something. At position 8 the learned column reads “no entry” and keeps reading it for 9 and 10, while sinusoidal carries on producing +0.989, +0.412 and −0.544 without difficulty.
That is the whole trade in four rows. A lookup table is defined exactly where it has rows and nowhere else, so a model trained with max_pos of 2048 meets a 2049 token input and has nothing to hand its attention layer. A formula is defined at every integer, so the same model with sinusoidal encoding produces something for position 2048, position 5000 and position 100000. Whether what it produces is useful that far out is a separate question, since the model has never seen those encodings during training, but it exists rather than failing.
The cost side is the last two lines. Eight positions of four dimensions is 32 floats here, and the realistic case of 2048 positions at 768 dimensions is 1,572,864 floats, which is 6144 kilobytes at single precision. That is a real allocation sitting in the parameter budget purely to hold position, and sinusoidal spends none of it.
Learned embeddings won on benchmarks by small margins for several years, which is why GPT-2 and BERT both use them. The length limit is what eventually pushed the field elsewhere.
22.7 Rotary Position Embeddings
RoPE takes a different route entirely. Instead of adding anything to the embedding, it rotates the query and key vectors by an angle proportional to their position, treating consecutive pairs of dimensions as coordinates in a plane and spinning each pair by its own frequency.
The reason to bother is what happens to the dot product afterwards. Rotating two vectors by angles that depend on their positions leaves a dot product that depends on the difference between those angles, and the difference between two positions is the distance between them. Relative position falls out of the arithmetic rather than being learned.
/* 119_Rope.c */
#include <stdio.h>
#include <math.h>
#define DIM 8 /* must be even */
/* Apply RoPE to a vector: rotate pairs of dimensions */
static void apply_rope(const float *x, int dim,
int pos, float *out)
{
int i;
for (i = 0; i < dim; i += 2) {
float freq = 1.0f / powf(10000.0f,
(float)i / dim);
float angle = pos * freq;
float cos_a = cosf(angle);
float sin_a = sinf(angle);
/* 2D rotation of dimensions (i, i+1) */
out[i] = x[i] * cos_a - x[i + 1] * sin_a;
out[i + 1] = x[i] * sin_a + x[i + 1] * cos_a;
}
}
int main(void)
{
/* A query and a key vector */
float q[DIM] = { 1.0f, 0.0f, 0.5f, 0.5f,
0.3f, 0.7f, 0.2f, 0.8f };
float k[DIM] = { 0.8f, 0.2f, 0.6f, 0.4f,
0.5f, 0.5f, 0.1f, 0.9f };
int i;
printf("Rotary Position Embeddings (RoPE):\n\n");
printf("Original query: [");
for (i = 0; i < DIM; i++)
printf("%.1f%s", q[i], i<DIM-1?", ":"");
printf("]\n\n");
/* Show how the query changes with position */
printf("Query rotated at different positions:\n");
int positions[] = { 0, 1, 2, 5, 10 };
int n_pos = 5;
for (int p = 0; p < n_pos; p++) {
float q_rot[DIM];
apply_rope(q, DIM, positions[p], q_rot);
printf(" pos %2d: [", positions[p]);
for (i = 0; i < DIM; i++)
printf("%+.2f%s", q_rot[i],
i<DIM-1?", ":"");
printf("]\n");
}
/* Show that dot product encodes relative
position */
printf("\nDot product of q(pos_q) and k(pos_k)\n");
printf(" relative distance matters, not the "
"absolute position\n\n");
printf(" pos_q pos_k dist dot_product\n");
int test_pairs[][2] = { {0, 0}, {0, 1}, {0, 2},
{5, 6}, {5, 7}, {10, 11} };
int n_tests = 6;
for (int t = 0; t < n_tests; t++) {
int pq = test_pairs[t][0],
pk = test_pairs[t][1];
float q_rot[DIM], k_rot[DIM];
apply_rope(q, DIM, pq, q_rot);
apply_rope(k, DIM, pk, k_rot);
float dp = 0;
for (i = 0; i < DIM; i++)
dp += q_rot[i] * k_rot[i];
printf(" %3d %3d %3d %+.4f\n",
pq, pk, pk - pq, dp);
}
printf("\nPairs the same distance apart, 0-1 "
"and 5-6 and 10-11,\n");
printf("give the same dot product wherever "
"they sit.\n");
printf("RoPE puts RELATIVE position straight "
"into the\n");
printf("attention score through rotation.\n");
return 0;
}
The upper block shows one query rotated to five different positions. At position 0 nothing moves, since the rotation angle is zero. By position 1 the first pair has swung from [1.00, 0.00] to [0.54, 0.84], and by position 10 it reads [−0.84, −0.54], well past a quarter turn. Look along any row and the later pairs barely change, with the last pair sitting at [0.20, 0.80] at every position shown, because their frequency is low enough that ten positions is almost nothing to them. Same frequency spreading as the sinusoidal encoding, applied by rotation rather than addition.
The lower table is the result that matters, and it is exact rather than approximate. The pair at positions 0 and 1 gives a dot product of +2.0133. The pair at 5 and 6 gives +2.0133. The pair at 10 and 11 gives +2.0133. Three different places in the sequence, one distance apart in each case, and the same number three times. Distance 2 does the same thing, with both (0,2) and (5,7) giving +1.2388.
Compare that to what an additive encoding offers. There the model receives absolute positions and has to learn that the gap between them is what matters, which it can do because of the linearity property from Section 22.4, but it has to learn it. With RoPE the attention score is a function of the gap by construction and there is nothing to learn.
Three practical consequences follow. RoPE applies to queries and keys only and never touches the values, so the content being retrieved is unrotated. It adds no parameters at all, since the rotation angles come from a formula. And it has no maximum position, for the same reason sinusoidal has none. Those three together are why LLaMA, Mistral and most models built since 2022 use it.
22.8 Comparing the Three
Nothing here needs computing, so the comparison is a table rather than a program.
| Sinusoidal | Learned | RoPE | |
|---|---|---|---|
| Extra parameters | none | max_pos * d_model | none |
| Extrapolates | yes | no | yes |
| Position type | absolute | absolute | relative |
| Applied to | embeddings | embeddings | Q and K |
| Used by | original paper | GPT-2, BERT | LLaMA, Mistral |
Read the parameters row against the extrapolation row and the learned column is paying twice, once in memory and once in flexibility. It earns that back only in benchmark scores, and by small margins.
The position type row is the one that separated RoPE from the other two. Sinusoidal and learned both hand the model an absolute location and rely on it to work out that relative distance is what matters. RoPE encodes the distance directly, which Section 22.7 confirmed by producing the same +2.0133 for three different pairs one position apart.
The applied to row explains why RoPE cannot simply be swapped in for the others. Sinusoidal and learned encodings are added to the input once, before the first layer, and every layer thereafter inherits the result. RoPE is applied inside the attention computation, to the queries and keys after projection, which means every layer applies it again and the values never carry position at all.
22.9 Key Takeaways
Attention is permutation equivariant, so without positional information it cannot distinguish “dog bites man” from “man bites dog”. Section 22.3 confirmed it by swapping two words and getting every attention row back unchanged.
The verb in that demonstration gave 0.32 to its subject and 0.32 to its object in both orderings, which is the failure at its clearest.
Sinusoidal encoding computes a vector per position from sine and cosine at falling frequencies. Section 22.4 measured the distance between consecutive positions at 0.9641 for every pair tested, which is what lets a model learn a displacement once and reuse it anywhere.
Because sin(a+b) and cos(a+b) expand into linear combinations, the encoding at p+k is a fixed linear function of the encoding at p depending only on k. Relative position is learnable rather than built in.
Positional vectors are added to embeddings rather than concatenated, which avoids doubling the width. The encoding is bounded to the same scale as the embeddings so neither signal swamps the other.
Learned embeddings allocate one trainable vector per position and stop dead at the end of the table. Section 22.6 showed positions 8 and beyond returning nothing while sinusoidal kept producing values.
The learned table costs max_pos * d_model parameters, which at 2048 positions and 768 dimensions is 1,572,864 floats or 6144 kilobytes. Sinusoidal and RoPE cost nothing.
RoPE rotates queries and keys by a position dependent angle, so the dot product becomes a function of the gap between positions. Section 22.7 measured +2.0133 for the pairs (0,1), (5,6) and (10,11), identical to four decimal places.
RoPE touches queries and keys only, never values, adds no parameters and has no maximum position. That combination is why it became the standard for modern models.
Sinusoidal and learned encodings enter once before the first layer. RoPE is applied inside every attention computation, which is why it cannot be swapped in as a drop-in replacement.
22.10 Exercises
Compute the sinusoidal encoding for positions 0 through 99 with d_model = 64. Print the encoding for dimensions 0 and 1 (fast frequency) vs dimensions 62 and 63 (slow frequency). How many positions before the slow dimension completes one full cycle?
Show that the dot product of two sinusoidal encodings at positions p and p+k is the same for any p (only k matters). This is the relative position property.
Implement RoPE and verify that the dot product of rotated q and k is invariant to absolute position: dot(rope(q, 0), rope(k, 3)) should equal dot(rope(q, 100), rope(k, 103)).
What happens if you use a very small d_model (e.g., 2) for sinusoidal encoding? At what point do two different positions get identical encodings?
Implement a hybrid: learned embeddings for the first 512 positions, sinusoidal extrapolation beyond that. This is a practical approach for models that need to handle long inputs occasionally.
RoPE rotates pairs of dimensions. If d_model is odd, one dimension has no pair. How would you handle this? (Most implementations require even d_model or d_head.)