LoRA
Low-Rank Adaptation for efficient fine-tuning
34.1 What You Will Learn
Back in Chapter 31 we established that fine-tuning is cheap relative to pretraining, and cheap is a relative word. Adapting a 7 billion parameter model still means computing a gradient for every one of those parameters and holding optimizer state alongside them. For Adam this is two further copies of everything. At 671 billion parameters that you might see in something like DeepSeek-V3, the arithmetic stops being a budget question and becomes an impossibility on any hardware a normal organization owns. Enter Low-Rank Adaptation usually shortened to LoRA. Low-Rank Adaptation is clever because it sidesteps the problem rather than solving it. The pretrained weights are frozen and never receive a gradient at all. Alongside each frozen matrix sits a pair of much smaller matrices whose product has the same shape, and only that pair is trained. A 4096 by 4096 matrix holds 16,777,216 numbers, while a rank 16 adapter for it holds 4096 times 16 twice over, which is 131,072, a reduction of exactly 128 times.
The point I’m getting at here and what I want you to learn in this chapter, is that the claim underneath is empirical rather than mathematical, and in this chapter we test it directly. LoRA works if the change a fine-tune wants to make to a weight matrix has low intrinsic rank, meaning it can be built from a handful of independent directions. Nothing guarantees that, and the whole method rests on it being true often enough in practice, where if you have been following along with the theme of this book is where it matters most.
This chapter builds the decomposition, the forward pass, the training loop across four ranks so the capacity limit is visible rather than asserted, the merge that removes the inference cost, and the parameter counts at deployed model sizes. Let’s dive into this chapter.
34.2 The Core Idea
The core idea is absurdly simple. A frozen weight matrix W keeps doing what it did and what we expect it to do as we’ve been exploring so far. Beside it sits a path through two smaller matrices, and the layer’s output is the sum of both. Look at this equation:
A has shape r by d_in and projects the input down to r dimensions, and B has shape d_out by r and projects back up. The rank r is small, typically 4 to 32, and it is the only knob controlling how much capacity the adapter has. W never receives a gradient. Look at this image and that may clear things up a bit.
Figure 34-1 puts the two arrangements side by side. Both end at the same place, a set of updated weights formed by adding something to the frozen pretrained ones, and both leave the pretrained block untouched. The only difference is what gets added. On the left it is a full-rank matrix the same shape as W. On the right it is the product of two narrow factors, drawn as the usual opposed pair, with the rank being the short dimension where they meet. The counts underneath use one attention matrix from LLaMA-7B, where d is 4096 and the rank is 8, giving sixteen million trainable numbers against sixty five thousand.
The reason for two matrices rather than one comes down to counting. A single trainable matrix of the same shape as W would hold exactly as many numbers as W does, so training it would cost precisely what full fine-tuning costs and nothing would have been gained. Routing the update through a narrow middle changes the arithmetic, because A holds r times d_in numbers and B holds d_out times r, so the total grows in proportion to d rather than to d squared. At d equal to 4096 those two growth rates differ by three orders of magnitude, and that gap is the entire method.
Note also what the arrangement rules out. The product of B and A has rank at most r no matter what values the two matrices take, which is a hard algebraic ceiling rather than a training difficulty. So this means the set of updates LoRA is capable of expressing is a small subset of all the updates that exist. Whether the particular update a given task needs happens to live inside that subset is an empirical question with no guaranteed answer. I won’t just leave you wondering though, as it is the question the next section puts numbers on.
34.3 What Low Rank Can and Cannot Represent
The chapter’s premise deserves testing rather than restating, since everything downstream from here depends on it. So the best way to build some intuition is to look at it for yourself. This program builds two 8 by 8 matrices with quite different structure. One is filled with unstructured random values and has full rank, which is what an arbitrary matrix looks like. The other is deliberately constructed as the product of an 8 by 2 and a 2 by 8, which makes it exactly rank 2 by construction. The program then fits a low-rank approximation to each of them at four increasing ranks and reports how much error survives in each case.
/* 164_Low_Rank.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#define N 8
#define MAXR 8
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
/* Fit B*A to target by gradient descent on the
squared error. An SVD would be exact and this is
enough to show how the error falls with rank. */
static float fit(const float target[N][N], int r,
int iters, float lr)
{
float A[MAXR][N], B[N][MAXR];
int i, j, k, it;
for (i = 0; i < r; i++)
for (j = 0; j < N; j++)
A[i][j] = (randf() * 2 - 1) * 0.5f;
for (i = 0; i < N; i++)
for (j = 0; j < r; j++)
B[i][j] = (randf() * 2 - 1) * 0.5f;
for (it = 0; it < iters; it++) {
for (i = 0; i < N; i++)
for (j = 0; j < N; j++) {
float p = 0;
for (k = 0; k < r; k++)
p += B[i][k] * A[k][j];
float e = p - target[i][j];
for (k = 0; k < r; k++) {
float b = B[i][k], a = A[k][j];
B[i][k] -= lr * e * a;
A[k][j] -= lr * e * b;
}
}
}
/* Relative error, ||BA - M|| over ||M|| */
float num = 0, den = 0;
for (i = 0; i < N; i++)
for (j = 0; j < N; j++) {
float p = 0;
for (k = 0; k < r; k++)
p += B[i][k] * A[k][j];
float d = p - target[i][j];
num += d * d;
den += target[i][j] * target[i][j];
}
return sqrtf(num / den);
}
int main(void)
{
float dense[N][N], lowrank[N][N];
float C[N][2], D[2][N];
int i, j, k, r;
srand(42);
/* A matrix with no structure at all */
for (i = 0; i < N; i++)
for (j = 0; j < N; j++)
dense[i][j] = (randf() * 2 - 1);
/* A matrix built to have rank 2, which is what a
fine-tuning update is claimed to look like */
for (i = 0; i < N; i++)
for (j = 0; j < 2; j++)
C[i][j] = (randf() * 2 - 1);
for (i = 0; i < 2; i++)
for (j = 0; j < N; j++)
D[i][j] = (randf() * 2 - 1);
for (i = 0; i < N; i++)
for (j = 0; j < N; j++) {
lowrank[i][j] = 0;
for (k = 0; k < 2; k++)
lowrank[i][j] += C[i][k] * D[k][j];
}
printf("Approximating %dx%d by B*A\n\n", N, N);
printf(" rank params vs %d dense err "
"rank2 err\n", N * N);
printf(" ---- ------ ----- --------- "
"---------\n");
for (r = 1; r <= 8; r *= 2) {
int p = 2 * N * r;
srand(7);
float e_dense = fit(dense, r, 4000, 0.02f);
srand(7);
float e_low = fit(lowrank, r, 4000, 0.02f);
printf(" %4d %6d %4.2fx %9.4f %10.4f\n",
r, p, (float)(N*N) / p, e_dense, e_low);
}
printf("\n The last column collapses at rank 2 "
"and\n");
printf(" stays there, because the target "
"really\n");
printf(" was rank 2 and nothing is left to "
"fit.\n");
printf(" The dense column falls slowly and "
"only\n");
printf(" reaches zero at full rank.\n\n");
printf(" That gap is the whole bet behind "
"LoRA.\n");
printf(" It works if a fine-tuning update "
"looks\n");
printf(" like the last column rather than the\n");
printf(" one before it, which is an empirical\n");
printf(" claim about training and not a "
"theorem.\n");
return 0;
}
Figure 34-2 plots the approximation error against rank for both targets. The two error columns behave completely differently and that difference is the chapter. Against the rank 2 target the error is 0.5616 at rank 1 and then collapses to 0.0000 at rank 2. This is because at that point the approximation can represent the target exactly and there is nothing left to fit. Against the unstructured target it falls slowly, look at it, it’s 0.8095 then 0.6479 then 0.3413, and reaches zero only at rank 8, which is full rank and no longer an approximation of anything.
The rank 8 entry in the last column may read as a small nonzero figure such as 0.0008 rather than a clean zero, and that is the fitting procedure rather than the mathematics. Approximating a rank 2 target with a rank 8 product is heavily overparameterized, so there are many exact solutions and gradient descent wanders among near-solutions instead of settling on one. A singular value decomposition would return zero. The point stands either way, since the error is three orders of magnitude below where the dense column sits at the same rank.
Read the parameter column beside them. Rank 2 costs 32 numbers against the 64 in the full matrix, a saving of exactly half, and rank 8 costs 128, which is twice the size of the matrix it is approximating. Low rank is only a saving while the rank stays well below the dimension, and at 8 by 8 there is barely any room for it to be one.
The bet LoRA makes is now stateable precisely.
It works if the update a fine-tune wants resembles the last column rather than the one before it. That is a claim about what gradient descent does to a pretrained network during adaptation, and it is supported by measurement rather than proof. The original LoRA paper found that updates were well approximated at surprisingly low rank across a range of tasks, and there is no theorem saying they must be.
One detail about the fitting. This program uses gradient descent to find B and A, and a singular value decomposition would give the optimal answer directly and in one step. The result would be the same shape, with error falling to zero exactly when the rank reaches the true rank of the target.
34.4 The Forward Pass
The forward pass adds the two paths together and there is nothing surprising in the arithmetic, so the thing to dwell on is how the adapter is initialized. A receives small random values in the ordinary way and B is set to exactly zero, which means their product is the zero matrix and the LoRA path contributes nothing whatsoever on the first forward pass. That asymmetry between the two matrices is deliberate and the next section explains why it has to be there.
/* 165_Lora_Forward.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define D_IN 6
#define D_OUT 6
#define RANK 2
typedef struct {
float W[D_OUT][D_IN];
/* frozen pretrained weights */
/* trainable: down-project */
float A[RANK][D_IN];
float B[D_OUT][RANK]; /* trainable: up-project */
float alpha; /* scaling factor */
}
LoRALayer;
static void lora_init(LoRALayer *l)
{
int i, j;
/* Pretrained weights (frozen, random for demo) */
for (i = 0; i < D_OUT; i++)
for (j = 0; j < D_IN; j++)
l->W[i][j] = (randf()*2-1) * 0.3f;
/* A gets small random values, B gets zeros */
for (i = 0; i < RANK; i++)
for (j = 0; j < D_IN; j++)
l->A[i][j] = (randf()*2-1) * 0.01f;
/* B is zero at the start, so the LoRA path
contributes nothing and the model behaves
exactly like the pretrained one. */
for (i = 0; i < D_OUT; i++)
for (j = 0; j < RANK; j++)
l->B[i][j] = 0.0f;
l->alpha = 1.0f;
}
static void lora_forward(const LoRALayer *l,
const float x[D_IN],
float out[D_OUT])
{
int i, j;
float scale = l->alpha / RANK;
/* Frozen path: W * x */
for (i = 0; i < D_OUT; i++) {
out[i] = 0;
for (j = 0; j < D_IN; j++)
out[i] += l->W[i][j] * x[j];
}
/* LoRA path: (alpha/r) * B * A * x */
float mid[RANK];
for (i = 0; i < RANK; i++) {
mid[i] = 0;
for (j = 0; j < D_IN; j++)
mid[i] += l->A[i][j] * x[j];
}
for (i = 0; i < D_OUT; i++)
for (j = 0; j < RANK; j++)
out[i] += scale * l->B[i][j] * mid[j];
}
int main(void)
{
LoRALayer l;
srand(42);
lora_init(&l);
float x[D_IN] = { 0.5f, -0.2f, 0.8f,
0.1f, -0.3f, 0.6f };
float out_before[D_OUT], out_after[D_OUT];
int i;
/* Before training, with B still zero */
lora_forward(&l, x, out_before);
/* Simulate training: set B to nonzero values */
for (i = 0; i < D_OUT; i++) {
l.B[i][0] = (randf()*2-1) * 0.1f;
l.B[i][1] = (randf()*2-1) * 0.1f;
}
lora_forward(&l, x, out_after);
printf("LoRA forward pass:\n\n");
printf(" output = W*x + (alpha/r) * B*A*x\n\n");
printf(" Before training, B=0 and LoRA idle\n "
"[");
for (i = 0; i < D_OUT; i++)
printf("%+.4f%s", out_before[i],
i<D_OUT-1?", ":"");
printf("]\n\n");
printf(" After training, B nonzero\n [");
for (i = 0; i < D_OUT; i++)
printf("%+.4f%s", out_after[i],
i<D_OUT-1?", ":"");
printf("]\n\n");
printf(" Difference (the LoRA adaptation):\n "
"[");
for (i = 0; i < D_OUT; i++)
printf("%+.4f%s",
out_after[i]-out_before[i],
i<D_OUT-1?", ":"");
printf("]\n\n");
printf(" B starts at zero, so the model is\n");
printf(" identical to the pretrained one at "
"the\n");
printf(" start. Only A and B ever move.\n\n");
int frozen = D_OUT * D_IN;
int trainable = RANK * D_IN + D_OUT * RANK;
printf(" Frozen params: %d\n", frozen);
printf(" Trainable params: %d\n", trainable);
printf(" Ratio: %.1f%%\n\n",
100.0f * trainable / frozen);
printf(" Two thirds is a terrible ratio and it\n");
printf(" is honest. At d=6 with r=2 the "
"adapter\n");
printf(" is nearly the size of what it adapts.\n");
printf(" LoRA only pays when d is large, since\n");
printf(" W grows as d squared while A and B "
"grow\n");
printf(" as d. Listing 5 has the real numbers.\n");
return 0;
}
Figure 34-3 shows the LoRA path before and after it becomes active. That initialization is not an arbitrary choice. Setting B to zero means the adapted model starts out numerically identical to the pretrained model, so fine-tuning begins from a known good state rather than from a perturbation of one. If both matrices were initialized randomly the model would be immediately worse than the checkpoint it started from and would spend its first steps recovering.
You should note that A cannot also be zero. If both were zero the gradient with respect to each would involve the other and would be zero as well, so nothing would ever move. One of the two has to be nonzero to break the symmetry, and it is A because that keeps the product zero.
The parameter counts at the bottom read 36 frozen against 24 trainable, a ratio of 66.7 percent, and the listing says plainly that this is a terrible advertisement. At six dimensions with rank 2 the adapter is nearly the size of the thing it adapts. That is not a flaw in the demonstration, it is a property of small matrices. It exists because W grows as d squared while A and B grow only as d. The two costs cross somewhere and everything useful about LoRA lives past the crossing, which we quantify at the end of the chapter.
34.5 Training Across Four Ranks
The task here is small enough to reason about completely, which is the point of choosing it. A four dimensional input vector must be mapped to its own reverse, so the target is a permutation matrix. W is frozen at random values and never moves, which means the adapter is responsible for supplying the entire difference between whatever W happens to do and what the permutation requires. This program runs that same setup four times at ranks 1 through 4 and reports what each of them manages.
/* 166_Lora_Train.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define D 4
#define R 4 /* the largest rank swept below */
typedef struct {
float W[D][D]; /* frozen */
float A[R][D]; /* trainable */
float B[D][R]; /* trainable */
}
LoRA;
static void lora_fwd(const LoRA *l, int r,
const float x[D], float out[D],
float mid[R])
{
int i, j;
for (i = 0; i < D; i++) {
out[i] = 0;
for (j = 0; j < D; j++)
out[i] += l->W[i][j] * x[j];
}
for (i = 0; i < r; i++) {
mid[i] = 0;
for (j = 0; j < D; j++)
mid[i] += l->A[i][j] * x[j];
}
for (i = 0; i < D; i++)
for (j = 0; j < r; j++)
out[i] += l->B[i][j] * mid[j];
}
static float train_step(LoRA *l, int r,
const float x[D],
const float target[D], float lr)
{
float out[D], mid[R];
int i, j, k;
lora_fwd(l, r, x, out, mid);
float loss = 0;
float d_out[D];
for (i = 0; i < D; i++) {
d_out[i] = 2.0f * (out[i] - target[i]);
loss += (out[i] - target[i])
* (out[i] - target[i]);
}
/* d_mid reads B, so it is computed before B is
updated. Differentiating through the new
values would give a wrong gradient without
any warning that it had. */
float d_mid[R];
for (i = 0; i < r; i++) {
d_mid[i] = 0;
for (k = 0; k < D; k++)
d_mid[i] += d_out[k] * l->B[k][i];
}
for (i = 0; i < D; i++)
for (j = 0; j < r; j++)
l->B[i][j] -= lr * d_out[i] * mid[j];
for (i = 0; i < r; i++)
for (j = 0; j < D; j++)
l->A[i][j] -= lr * d_mid[i] * x[j];
/* W is never touched */
return loss;
}
int main(void)
{
/* Reverse the order of a 4-vector. W is frozen
and random, so the LoRA update has to supply
the whole difference between W and the
permutation, which is a general 4x4 matrix. */
float inputs[D][D] = {
{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1,
0}, {0, 0, 0, 1}
};
float targets[D][D] = {
{0, 0, 0, 1}, {0, 0, 1, 0}, {0, 1, 0,
0}, {1, 0, 0, 0}
};
float W0[D][D];
int i, j, r, ep, p;
srand(42);
for (i = 0; i < D; i++)
for (j = 0; j < D; j++)
W0[i][j] = (randf()*2-1) * 0.3f;
printf("LoRA at four ranks, W frozen "
"throughout\n\n");
printf(" rank trained final loss worst "
"output\n");
printf(" ---- ------- ---------- "
"------------\n");
for (r = 1; r <= D; r++) {
LoRA l;
for (i = 0; i < D; i++)
for (j = 0; j < D; j++)
l.W[i][j] = W0[i][j];
srand(7);
for (i = 0; i < R; i++)
for (j = 0; j < D; j++)
l.A[i][j] = (randf()*2-1) * 0.5f;
/* B starts at zero so the adapter begins as
a no-op and the model is unchanged */
for (i = 0; i < D; i++)
for (j = 0; j < R; j++) l.B[i][j] = 0;
float loss = 0;
for (ep = 0; ep < 4000; ep++) {
loss = 0;
for (p = 0; p < D; p++)
loss += train_step(&l, r, inputs[p],
targets[p], 0.02f);
loss /= D;
}
/* How far is the worst position from its
target across all four test vectors? */
float worst = 0;
for (p = 0; p < D; p++) {
float out[D], mid[R];
lora_fwd(&l, r, inputs[p], out, mid);
for (i = 0; i < D; i++) {
float e = fabsf(out[i] - targets[p][i]);
if (e > worst) worst = e;
}
}
printf(" %4d %7d %10.4f %12.4f\n",
r, 2 * D * r, loss, worst);
}
printf("\n Rank 4 drives the loss to nothing. "
"The\n");
printf(" lower ranks cannot, and the reason is\n");
printf(" structural rather than a matter of "
"more\n");
printf(" training. W is frozen, so the adapter\n");
printf(" must supply the entire difference\n");
printf(" between W and the permutation, and "
"that\n");
printf(" difference is a general 4x4 matrix of\n");
printf(" rank 4. A rank-2 update cannot reach "
"it\n");
printf(" however long it runs.\n\n");
printf(" At this size rank 4 also costs 32\n");
printf(" parameters against the 16 in W, so "
"LoRA\n");
printf(" is a loss here. It pays only when d "
"is\n");
printf(" large and the useful rank stays "
"small.\n");
return 0;
}
Figure 34-4 runs the same task at four ranks, where the capacity limit becomes visible. The final loss falls monotonically through 0.7871, 0.3748, 0.1533 and reaches exactly 0.0000 only at rank 4. The worst individual output tells a blunter version of the same story, sitting at 1.0284 for rank 1 and still above 0.71 at ranks 2 and 3 before dropping to nothing at rank 4.
Those middle two entries are worth a second look. The loss more than halves between rank 2 and rank 3, from 0.3748 to 0.1533, while the worst single output barely moves, going from 0.7144 to 0.7123. Average error improved considerably and the hardest case did not, which is a pattern worth recognizing, since a mean can hide a position the model still gets badly wrong.
That pattern is not a training failure at the lower ranks and no amount of extra epochs would change it. W is frozen at random values, so the adapter must produce the entire matrix difference between W and the target permutation, and the difference between a random matrix and a permutation is a general 4 by 4 matrix of full rank. A rank 2 update has at most rank 2 and simply cannot reach it, so the residual is a hard floor set by the architecture rather than by the optimizer.
An earlier version of this listing trained at rank 2 only, plateaued at a loss of 0.4354, and concluded that the adapter had learned to reverse the order. It had not. One of the four test vectors came out with its top two outputs at 0.31 and 0.30, which is a coin flip rather than a correct answer, and the claim was contradicted by the numbers printed directly above it.
The rank column is also the answer to exercise 1, and the honest reading of the last row is unflattering. Rank 4 works and costs 32 trained parameters against the 16 sitting frozen in W, so at this size LoRA is strictly worse than fine-tuning the matrix directly. Every argument for the method is an argument about scale.
34.6 Merging
Keeping the adapter separate costs an extra matrix multiply on every single forward pass, which is negligible during training and becomes a real cost at inference, where the same weights run many millions of times. The fix requires no approximation and no retraining. Both paths are linear maps and the layer adds them, so the sum of two linear maps is itself a linear map and the whole arrangement collapses into one matrix.
The product of B and A has the same shape as W, so adding the two gives one matrix that behaves exactly like the pair did and can replace W outright. The factor alpha over r is the scaling LoRA applies to the adapter output during training, and it has to be carried into the merge or the merged model will not match the model that was trained. Nothing here is an approximation, since both paths are linear and the sum of two linear maps is a linear map, which is why merging costs no accuracy and cannot be undone by rounding.
/* 167_Merge.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define D 4
#define R 2
int main(void)
{
float W[D][D], A[R][D], B[D][R];
int i, j, k;
srand(42);
for (i = 0; i < D; i++)
for (j = 0; j < D; j++)
W[i][j] = (randf()*2-1) * 0.3f;
for (i = 0; i < R; i++)
for (j = 0; j < D; j++)
A[i][j] = (randf()*2-1) * 0.1f;
for (i = 0; i < D; i++)
for (j = 0; j < R; j++)
B[i][j] = (randf()*2-1) * 0.1f;
/* Compute B*A */
float BA[D][D];
for (i = 0; i < D; i++)
for (j = 0; j < D; j++) {
BA[i][j] = 0;
for (k = 0; k < R; k++)
BA[i][j] += B[i][k] * A[k][j];
}
/* Merge: W_new = W + BA */
float W_merged[D][D];
for (i = 0; i < D; i++)
for (j = 0; j < D; j++)
W_merged[i][j] = W[i][j] + BA[i][j];
/* Verify: W*x + B*A*x == W_merged*x */
float x[D] = { 0.5f, -0.2f, 0.8f, 0.3f };
float out_separate[D] = {0}, out_merged[D] = {0};
/* Separate: W*x + B*(A*x) */
for (i = 0; i < D; i++)
for (j = 0; j < D; j++)
out_separate[i] += W[i][j] * x[j];
float mid[R] = {0};
for (i = 0; i < R; i++)
for (j = 0; j < D; j++)
mid[i] += A[i][j] * x[j];
for (i = 0; i < D; i++)
for (j = 0; j < R; j++)
out_separate[i] += B[i][j] * mid[j];
/* Merged: W_merged*x */
for (i = 0; i < D; i++)
for (j = 0; j < D; j++)
out_merged[i] += W_merged[i][j] * x[j];
printf("LoRA weight merging:\n\n");
printf(" W_merged = W + B*A\n\n");
printf(" Separate: [");
for (i = 0; i < D; i++)
printf("%+.4f%s", out_separate[i],
i<D-1?", ":"");
printf("]\n");
printf(" Merged: [");
for (i = 0; i < D; i++)
printf("%+.4f%s", out_merged[i],
i<D-1?", ":"");
printf("]\n\n");
float diff = 0;
for (i = 0; i < D; i++) {
float d = out_separate[i] - out_merged[i];
diff += d * d;
}
printf(" Difference: %.10f\n\n", sqrtf(diff));
printf(" After merging, inference is one "
"matrix\n");
printf(" multiply rather than two, so the\n");
printf(" adapter costs nothing to serve.\n\n");
printf(" Or keep it separate and swap adapters\n");
printf(" base + LoRA_medical = medical\n");
printf(" base + LoRA_coding = coding\n");
printf(" base + LoRA_legal = legal\n");
printf(" One frozen base, many behaviours.\n");
return 0;
}
Figure 34-5 merges the adapter into the frozen weights. The two output vectors agree to four decimal places and the reported difference between them is around 0.00000003, which is the accumulated rounding of single-precision arithmetic rather than any approximation in the method. Merging is exact in the mathematical sense, because the sum of two linear maps is a linear map with no remainder, and nothing at all is discarded in forming it.
The consequence is that LoRA’s inference overhead is optional. Merge and the adapter is invisible, with the served model being an ordinary weight matrix that costs precisely what the original cost. That is a meaningful difference from adapter methods that insert extra layers into the network, since those cannot be folded away and their cost is permanent.
Keeping the adapter unmerged buys something else though, and the closing lines point at it. A single frozen base can be paired with any number of small adapters, swapped at request time, so one set of large weights serves a medical variant, a coding variant and a legal variant. The adapters are megabytes against the base model’s gigabytes, which changes what it costs to offer many specialized models rather than one.
34.7 The Numbers at Real Scale
Every listing so far has run on matrices small enough that LoRA loses outright, which was honest but leaves the method looking pointless. This section is where it wins, and nothing new is introduced to make that happen. The reason is entirely the difference in growth rates we set out at the start of the chapter, applied at dimensions where that difference has room to matter.
/* 168_Savings.c */
#include <stdio.h>
int main(void)
{
printf("LoRA parameter savings for real "
"models:\n\n");
struct {
const char *name;
int d_model;
int n_layers;
int n_matrices;
/* W_Q, W_K, W_V, W_O per layer */
}
models[] = {
{ "LLaMA-7B", 4096, 32, 4 },
{ "LLaMA-70B", 8192, 80, 4 },
{ "DeepSeek-V3", 7168, 61, 4 },
};
int n = 3, i;
int ranks[] = { 4, 8, 16, 32 };
int n_ranks = 4;
for (i = 0; i < n; i++) {
int d = models[i].d_model;
int L = models[i].n_layers;
int M = models[i].n_matrices;
long long full = (long long)d * d * L * M;
printf(" %s, d=%d, %d layers, %d attention "
"matrices\n",
models[i].name, d, L, M);
printf(" Full fine-tune: %lld params "
"(%.1fM)\n", full, full/1e6);
for (int r = 0; r < n_ranks; r++) {
int rank = ranks[r];
long long lora =
(long long)(d * rank + rank * d)
* L * M;
printf(" LoRA rank=%2d: %lld params "
"(%.1fM, %.0fx less)\n",
rank, lora, lora/1e6,
(float)full / lora);
}
printf("\n");
}
printf(" At rank 16, LLaMA-7B trains 16.8M "
"rather\n");
printf(" than 2.1B, which is the 128x in the "
"table\n");
printf(" above and fits on one consumer "
"card.\n\n");
printf(" The frozen weights still have to be "
"held\n");
printf(" in memory, and that is the real floor "
"on\n");
printf(" what LoRA can do. What it removes is "
"the\n");
printf(" gradient and the optimizer state, "
"which\n");
printf(" for Adam is two more copies of every\n");
printf(" parameter being trained.\n");
return 0;
}
Figure 34-6 gives what LoRA costs at three real model sizes. If we want a real world number you can consider LLaMA 7B. LLaMA 7B’s attention matrices come to 2,147,483,648 parameters across 32 layers, and a rank 16 adapter for all of them comes to 16,777,216, which is a reduction of 128 times. At rank 4 the reduction is 512 times and at rank 32 it is 64, the factor halving each time the rank doubles, exactly as the linear growth in r predicts.
An earlier version of the closing text claimed rank 16 trained 67M parameters for a 31 times reduction, which contradicted the table printed immediately above it in both numbers. The table was right. The same wrong figure appeared in the chapter’s takeaways, and the opening section separately described a 128 times reduction as 122 times.
What LoRA actually removes is worth being a bit precise about, because the reduction factor overstates the memory saving. The frozen weights still have to be resident, since the forward pass needs them, so the base model’s memory footprint is untouched. What disappears is the gradient for those weights and the optimizer state attached to them, and for Adam that state is two further values per parameter. Full fine-tuning of a 7B model therefore needs something like four copies of the parameters in memory while LoRA needs one copy plus four copies of something 128 times smaller.
That is the difference between requiring a cluster and requiring a single card, and it is why the technique spread as fast as it did. It also explains the shape of the ecosystem, where thousands of published adapters share a handful of base models, each one a small file that means nothing on its own.
34.8 Key Takeaways
LoRA freezes the pretrained weights and trains a pair of small matrices whose product is added to the frozen output. Only A and B receive gradients.
Parameters grow as d for the adapter and as d squared for the matrix it adapts, which is the whole source of the saving and the reason LoRA loses on small matrices.
We measured that directly. Against a target that really was rank 2 the error collapsed to 0.0000 at rank 2, and against an unstructured target it fell only from 0.8095 to 0.3413 by rank 4 and reached zero only at full rank.
The method therefore rests on an empirical claim, that fine-tuning updates have low intrinsic rank in practice. That is measured rather than proved.
B is initialized to zero so the adapter starts as a no-op and the model is numerically identical to the checkpoint. A cannot also be zero, since both being zero would leave every gradient at zero and nothing would ever move.
We trained the same task at four ranks, reaching losses of 0.7871, 0.3748, 0.1533 and exactly 0.0000. The lower ranks fail structurally, since a frozen W means the adapter must supply a full-rank difference that a low-rank product cannot express.
More training does not help at insufficient rank. The floor is set by the architecture.
Merging is exact. We measured a difference of around 0.00000003 between the separate and merged paths, which is single-precision rounding, so the inference overhead is optional and disappears entirely on request.
Unmerged adapters can be swapped against one frozen base, so a single set of large weights serves many specializations, each adapter being megabytes against the base model’s gigabytes.
At rank 16 LLaMA-7B’s attention adapters come to 16.8M parameters against 2,147.5M, a reduction of 128 times. Doubling the rank halves the factor.
The reduction factor overstates the memory saving, because the frozen weights still have to be held. What LoRA removes is the gradient and the optimizer state, which for Adam is two more copies per trained parameter.
At six dimensions with rank 2 the adapter was 66.7 percent the size of what it adapted, and at four dimensions rank 4 cost 32 parameters against 16. Every argument for LoRA is an argument about scale.
34.9 Exercises
Train LoRA with rank 1. How much does quality degrade compared to rank 2 and rank 4?
What happens if you initialize B with random values instead of zeros? The model starts different from the pretrained model. Compare training convergence.
Apply LoRA to only the Q and V projections (not K and O). This is a common configuration. How does it compare to applying LoRA to all four?
Implement QLoRA: quantize the frozen weights to 4-bit and keep LoRA in full precision. This reduces memory further while maintaining adaptation quality.
Train two different LoRA adapters on two different tasks. Verify that merging both into the base model hurts performance on each individual task (interference).
The alpha/r scaling factor controls the magnitude of the LoRA contribution. Try alpha = r (scaling factor = 1.0), alpha = 2*r, and alpha = r/2. How does it affect convergence?