Quantization
Reducing precision to shrink models and speed up inference
35.1 What You Will Learn
In this chapter you will learn about quantization. Before we even get into what it is, I want you to consider this. A model holding seven billion parameters (a 7B model) in 32-bit floating point occupies twenty eight gigabytes before a single activation is allocated. Halving the width to sixteen bits brings that figure down to fourteen gigabytes, and halving it again brings the same model down to seven. At four bits per weight the storage requirement falls to three and a half gigabytes, which fits on hardware like a small laptop, tablet or smartphone that most readers already own. The case for the mechanism has been made and the name we give to the process is Quantization. Quantization is the transformation that makes reductions in memory possible, replacing high precision floating point weights with low precision integers drawn from a small fixed grid. If you dabbled with quantized models before then you’ll know that while quantization allows you to run powerful models with less compute, there is a tradeoff to make. Every bit you take away discards information the network was trained with, so the question running through this chapter is how much accuracy leaves along with it. This will be useful to you when you realize that some people run models on 1-bit quantization and the tradeoff in real world deployment is do you use a smaller model that’s less quantized (or less lobotomized as some practitioners say) or a large model that’s been heavily quantized for your application.
In this chapter we will work through understanding what quantization is by building four programs. The four programs you will work on build the standard schemes in order of increasing sophistication and measure what each one costs. Absmax quantization scales an entire tensor by its largest magnitude and provides the baseline that every later refinement improves on. Group quantization divides the tensor into small blocks with independent scales, so that one extreme weight cannot ruin the resolution available to its neighbors. Quantized matrix multiply asks where the promised speed is supposed to come from and measures whether a portable scalar loop actually delivers any of it. Four bit packing then pushes compression to its practical limit and lets you watch the error grow into something you can no longer ignore. After finishing this chapter, as usual the aim is that you will build intuition for quantization and understand the tradeoffs when deploying.
35.2 Number Representations
You probably already know this by heart, or maybe it’s your first time encountering it. Either way number representations are something I want to review with you before we move forward. So as you may know, floating point formats store a sign, an exponent and a mantissa, which lets a single 32-bit value cover a range that a fixed point format can’t approach. The exponent buys that enormous dynamic range while the mantissa determines how finely the format resolves values within it. Integer formats abandon dynamic range completely, and spread a fixed number of evenly spaced levels across whatever interval you choose for them. So an 8-bit signed integer offers two hundred and fifty six levels and nothing more, so the interval it covers has to be chosen with some care. Quantization is the pair of mappings that carries floating point values onto that grid and back again. What we will work on together is an investigation into what the round trip costs. Take a look at this table:
| Format | Bits | Approximate range | 7B model | Reduction |
|---|---|---|---|---|
| float32 | 32 | 3.4e38 | 28.0 GB | 1.0x |
| float16 | 16 | 65504 | 14.0 GB | 2.0x |
| bfloat16 | 16 | 3.4e38 | 14.0 GB | 2.0x |
| FP8 E4M3 | 8 | 448 | 7.0 GB | 4.0x |
| int8 | 8 | 127 | 7.0 GB | 4.0x |
| int4 | 4 | 7 | 3.5 GB | 8.0x |
What we can get from this table is that two of these formats occupy sixteen bits apiece and differ only in how they divide those bits between exponent and mantissa. The bfloat16 format keeps the full float32 exponent and surrenders mantissa bits to pay for it, which trades precision for reach. That trade matters during training because gradients span many orders of magnitude, and a gradient that underflows to zero does more damage than one that is merely imprecise. The ingenious thing that DeepSeek-V3 did is that it pushes further still and trains in 8-bit floating point, an approach widely considered impractical until the fine grained scaling described later in this chapter made it viable. Inference presents a different problem, since the weights are frozen and integer formats tend to win on size and speed together.
35.3 Absmax Quantization
The simplest scheme finds the largest magnitude anywhere in the tensor and stretches the integer range to cover exactly that much. Dividing by one hundred and twenty seven rather than one hundred and twenty eight keeps the mapping symmetric about zero and wastes a single level at the negative end. That wasted level buys something valuable, because a symmetric mapping removes the zero point offset from every calculation downstream. This matters enormously once you reach the matrix multiply. Rounding to nearest completes the forward direction and multiplying by the same scale reverses it, leaving an error bounded by half a scale step in the worst case. That bound is the whole theory of the method, and everything which follows is an argument about how to make the scale step smaller. Take a look at this pair of equations:
The scale s is the only thing stored alongside the integers, one float for the whole tensor. Each weight x_i becomes the integer q_i by dividing through by s and rounding, and reading it back means multiplying by s again to get the approximation x-hat_i. Dividing the largest magnitude by 127 rather than 128 keeps the range symmetric, so a weight and its negation quantize to opposite integers.
Figure 35-1 runs those two lines on the vector the program uses. The largest magnitude is 1.204, highlighted in the top row, and dividing it by 127 gives the scale printed above. Every value is then divided by that scale and rounded, which is the middle row, and multiplying back by it gives the bottom row. Two things are worth reading off the picture. Firstly, the largest value maps to exactly 127, using the integer range fully, and the recovered numbers differ from the originals in the third decimal at worst. Secondly, that worst case is bounded by half a step, since rounding cannot be wrong by more than that, and the step here is 0.0095. This is enough theory though, so let’s put some instructions to the machine then run our autopsy.
/* 169_Absmax.c */
#include <stdio.h>
#include <math.h>
#define N 8
/* Quantize float array to int8 using absmax */
static void quantize_absmax(const float *x,
signed char *q,
float *scale, int n)
{
/* Find max absolute value */
float absmax = 0;
int i;
for (i = 0; i < n; i++)
if (fabsf(x[i]) > absmax)
absmax = fabsf(x[i]);
*scale = absmax / 127.0f;
/* Quantize: round(x / scale) */
for (i = 0; i < n; i++) {
float v = x[i] / (*scale);
if (v > 127) v = 127;
if (v < -127) v = -127;
q[i] = (signed char)(v
+ (v >= 0 ? 0.5f : -0.5f));
}
}
/* Dequantize: multiply by scale */
static void dequantize(const signed char *q,
float scale,
float *out, int n)
{
int i;
for (i = 0; i < n; i++)
out[i] = q[i] * scale;
}
int main(void)
{
float x[N] = { 0.532f, -0.187f, 1.204f, -0.023f,
0.891f, -0.645f, 0.078f, -1.105f };
signed char q[N];
float scale, x_hat[N];
int i;
printf("Absmax int8 quantization:\n\n");
printf(" Original: [");
for (i = 0; i < N; i++)
printf("%+.3f%s", x[i], i<N-1 ? ", " : "");
printf("]\n");
quantize_absmax(x, q, &scale, N);
printf(" Quantized: [");
for (i = 0; i < N; i++)
printf("%4d%s", q[i], i<N-1 ? ", " : "");
printf("]\n");
printf(" Scale: %.6f\n", scale);
dequantize(q, scale, x_hat, N);
printf(" Dequantized: [");
for (i = 0; i < N; i++)
printf("%+.3f%s", x_hat[i], i<N-1 ? ", " : "");
printf("]\n\n");
/* Compute error */
float max_err = 0, sum_sq = 0;
for (i = 0; i < N; i++) {
float err = fabsf(x[i] - x_hat[i]);
if (err > max_err) max_err = err;
sum_sq += err * err;
}
printf(" Max error: %.6f\n", max_err);
printf(" RMS error: %.6f\n", sqrtf(sum_sq / N));
printf(" Memory: %d bytes, was %lu, %.1fx\n",
N * 1 + 4, /* int8 plus one float scale */
N * sizeof(float),
(float)(N * sizeof(float)) / (N + 4));
printf(" The scale is 4 bytes of fixed "
"overhead.\n");
printf(" Across a 128 value group it costs "
"%.1f%%\n",
100.0f * 4 / (128 + 4));
printf(" and the ratio reaches %.2fx.\n",
128.0f * 4 / (128 + 4));
return 0;
}
Figure 35-2 measures the error and the memory cost against the original float32 storage. On running our program we realize the eight element vector reaches a largest magnitude of 1.204, which gives a scale of 0.009480 and a worst case error of 0.004197 across the whole vector. That sits under half a scale step of 0.004740, and the root mean square error is 0.002417. That number is one fifth of one percent of the largest weight present! The memory line teaches more than the error line does, because twelve bytes against thirty two is a ratio of 2.7 rather than the fourfold reduction the format promised, hence why the emphasis is on testing rather than theory. Four of those twelve bytes are the scale itself, fixed overhead that a mere eight values have no hope of amortizing. If we spread the same scale across a group of one hundred and twenty eight values, and it costs three percent, that brings the ratio to 3.88 and the promise back into view.
The weakness of the scheme becomes visible the moment a tensor contains one value far larger than the rest of them. A vector holding 0.01, 0.02, 0.01 and 10.0 produces a scale of 0.0787. The three small values divide down to 0.13, 0.25 and 0.13 before rounding. Because all three of them round to zero, the quantized tensor preserves the outlier perfectly and destroys every other value it was asked to carry. However, this is not a contrived failure, because attention and feed forward layers in trained transformers reliably hold a handful of weights, and these handful of weights are an order of magnitude above their neighbors. The remedy is to stop allowing one number to set the scale for all the others.
35.4 Group Quantization
What I want us to turn our attention to now is group quantization. Dividing the tensor into contiguous groups, and giving each group its own scale, confines the damage an outlier can do to the group that happens to contain it. The cost is one extra float for every group, which at a group size of one hundred and twenty eight works out to three percent of overhead. If we examine DeepSeek-V3, we will see that it adopts exactly this structure, quantizing activations in tiles of one by one hundred and twenty eight, and weights in blocks of one hundred and twenty eight squared. The program below uses a group size of eight so the arithmetic stays visible on the page, and it plants a single outlier of 10.0 inside the first group. Both schemes then quantize the same data, which makes the comparison between them exact rather than merely indicative. Before we get into the program though, take a look at this:
G_g is the set of indices belonging to group g, so the maximum is taken over that group alone rather than the whole tensor. Everything else is the absmax rule from the previous section applied independently to each block. The tensor now carries one float per group instead of one float in total, which is the memory the method spends to stop an outlier in one block from setting the resolution for every other block. With that being said, let’s get into our program.
/* 170_Group_Quant.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N 32
#define GROUP_SIZE 8
static void quantize_group(const float *x,
signed char *q,
float *scales, int n,
int group_size)
{
int g, i;
int n_groups = n / group_size;
for (g = 0; g < n_groups; g++) {
int offset = g * group_size;
float absmax = 0;
for (i = 0; i < group_size; i++)
if (fabsf(x[offset + i]) > absmax)
absmax = fabsf(x[offset + i]);
scales[g] = absmax / 127.0f;
if (scales[g] < 1e-10f) scales[g] = 1e-10f;
for (i = 0; i < group_size; i++) {
float v = x[offset + i] / scales[g];
if (v > 127) v = 127;
if (v < -127) v = -127;
q[offset + i] = (signed char)
(v + (v >= 0 ? 0.5f : -0.5f));
}
}
}
static void dequantize_group(const signed char *q,
const float *scales,
float *out, int n,
int group_size)
{
int g, i;
int n_groups = n / group_size;
for (g = 0; g < n_groups; g++) {
int offset = g * group_size;
for (i = 0; i < group_size; i++)
out[offset + i] = q[offset + i] * scales[g];
}
}
int main(void)
{
float x[N];
signed char q[N];
float scales[N / GROUP_SIZE];
float x_hat[N];
int i, g;
srand(42);
/* Create data with an outlier in one group */
for (i = 0; i < N; i++)
x[i] = (randf()*2-1) * 0.5f;
x[3] = 10.0f; /* outlier */
printf("Group quantization (group_size=%d):\n\n",
GROUP_SIZE);
/* Per-tensor quantization */
signed char q_tensor[N];
float tensor_scale;
float absmax = 0;
for (i = 0; i < N; i++)
if (fabsf(x[i]) > absmax) absmax = fabsf(x[i]);
tensor_scale = absmax / 127.0f;
for (i = 0; i < N; i++) {
float v = x[i] / tensor_scale;
q_tensor[i] = (signed char)
(v + (v >= 0 ? 0.5f : -0.5f));
}
/* Group quantization */
quantize_group(x, q, scales, N, GROUP_SIZE);
dequantize_group(q, scales, x_hat, N, GROUP_SIZE);
/* Compute errors for both */
float err_tensor = 0, err_group = 0;
for (i = 0; i < N; i++) {
float deq_t = q_tensor[i] * tensor_scale;
err_tensor += (x[i] - deq_t) * (x[i] - deq_t);
err_group += (x[i] - x_hat[i])
* (x[i] - x_hat[i]);
}
printf(" Per-tensor RMS error: %.6f\n",
sqrtf(err_tensor / N));
printf(" Group (g=%d) RMS error: %.6f\n",
GROUP_SIZE, sqrtf(err_group / N));
printf(" Improvement: %.1fx\n\n",
sqrtf(err_tensor) / sqrtf(err_group));
/* Where the error lands, group by group */
printf(" grp absmax scale tensor group\n");
for (g = 0; g < N / GROUP_SIZE; g++) {
int o = g * GROUP_SIZE;
float et = 0, eg = 0;
for (i = 0; i < GROUP_SIZE; i++) {
float dt = q_tensor[o + i] * tensor_scale;
et += (x[o + i] - dt) * (x[o + i] - dt);
eg += (x[o + i] - x_hat[o + i])
* (x[o + i] - x_hat[o + i]);
}
printf(" %3d %8.4f %.6f %.4f %.4f\n", g,
scales[g] * 127.0f, scales[g],
sqrtf(et / GROUP_SIZE),
sqrtf(eg / GROUP_SIZE));
}
printf("\n Group 0 holds the outlier x[3]=10.0 "
"so\n");
printf(" its scale equals the tensor scale and "
"it\n");
printf(" gains nothing. The damage stops "
"there.\n\n");
printf(" DeepSeek-V3 uses:\n");
printf(" 1x128 tiles for activations\n");
printf(" 128x128 blocks for weights\n");
printf(" FP8 (E4M3) format instead of int8\n");
printf(" Online scales computed on the fly\n");
return 0;
}
Figure 35-3 applies both schemes to the same data and breaks the error out group by group to show where it accumulates. The aggregate improvement of 1.8 understates the method rather badly, and the per group table is what explains the discrepancy. Group zero contains the outlier and therefore computes a group scale identical to the tensor scale, so its error is unchanged to four decimal places. The remaining three groups watch their error fall from between 0.016 and 0.024 down to roughly 0.001. This corresponds to an improvement of sixteen to twenty times depending on the group. Averaging across all four mixes, one group that gained nothing whatsoever, with three that gained close to twenty. However, the mean arrives at a number describing neither of them. The honest reading is that group quantization never repairs an outlier, what it does is it quarantines one. This is important to keep in mind because the size of the quarantine is the only parameter you control in this scenario.
35.5 Quantized Matrix Multiply
You know it was coming so here it is, quantized matrix multiply. Shrinking the weights accounts for only half the usual argument for quantizing, because the arithmetic is supposed to get cheaper once both operands are eight bit integers. However, there are some other things we need to consider. A row of quantized weights multiplied against a quantized vector accumulates into a 32-bit integer. I want to state that this is with no floating point work anywhere in the inner loop. The two scales stay constant across the entire dot product, so they factor out of the summation and apply once to the finished accumulator. Keeping the dequantization outside the loop is the structural requirement of the whole approach, and it is easy to discard by converting back to float on every multiply. The program times both paths, over two hundred thousand repetitions rather than asserting which of them ought to win. Look at this equation:
Notice the two scales sit outside the sum, which is the entire point of the arrangement. Inside it the products W-q times x-q are integer multiplications accumulated into an integer, so no floating point arithmetic happens until the row is finished. The superscript on s_w indicates a per row weight scale, while s_x is one scale for the whole input vector, and multiplying the accumulated integer by both at the end recovers the float result.
/* 171_Quantized_Matmul.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
#include <time.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define M 64
#define K 64
#define N_VEC 64
static void quantize_vec(const float *x,
signed char *q,
float *scale, int n)
{
float absmax = 0;
int i;
for (i = 0; i < n; i++)
if (fabsf(x[i]) > absmax) absmax = fabsf(x[i]);
*scale = absmax / 127.0f;
if (*scale < 1e-10f) *scale = 1e-10f;
for (i = 0; i < n; i++) {
float v = x[i] / (*scale);
if (v > 127) v = 127;
if (v < -127) v = -127;
q[i] = (signed char)(v
+ (v >= 0 ? 0.5f : -0.5f));
}
}
int main(void)
{
/* A weight matrix and input vector */
float W[M][K], x[K];
int i, j;
srand(42);
for (i = 0; i < M; i++)
for (j = 0; j < K; j++)
W[i][j] = (randf()*2-1) * 0.5f;
for (j = 0; j < K; j++)
x[j] = (randf()*2-1) * 0.5f;
/* Float32 matmul */
float out_f32[M];
for (i = 0; i < M; i++) {
out_f32[i] = 0;
for (j = 0; j < K; j++)
out_f32[i] += W[i][j] * x[j];
}
/* Quantize W rows and x, multiply in int32,
then scale back to float */
signed char W_q[M][K], x_q[K];
float w_scales[M], x_scale;
for (i = 0; i < M; i++)
quantize_vec(W[i], W_q[i], &w_scales[i], K);
quantize_vec(x, x_q, &x_scale, K);
float out_q[M];
for (i = 0; i < M; i++) {
int acc = 0; /* int32 accumulator */
for (j = 0; j < K; j++)
acc += (int)W_q[i][j] * (int)x_q[j];
out_q[i] = acc * w_scales[i] * x_scale;
}
/* Compare */
float max_err = 0, rms_err = 0;
for (i = 0; i < M; i++) {
float err = fabsf(out_f32[i] - out_q[i]);
if (err > max_err) max_err = err;
rms_err += err * err;
}
rms_err = sqrtf(rms_err / M);
printf("Quantized matrix multiply "
"(%dx%d):\n\n", M, K);
printf(" First 8 outputs:\n");
printf(" pos float32 int8_quant error\n");
printf(" --- ---------- ---------- -----\n");
for (i = 0; i < 8; i++)
printf(" %2d %+.4f %+.4f %.4f\n",
i, out_f32[i], out_q[i],
fabsf(out_f32[i] - out_q[i]));
printf("\n Max error: %.6f\n", max_err);
printf(" RMS error: %.6f\n", rms_err);
float scale_out = 0;
for (i = 0; i < M; i++)
if (fabsf(out_f32[i]) > scale_out)
scale_out = fabsf(out_f32[i]);
printf(" Max error over largest output: %.2f%%\n",
max_err / scale_out * 100);
printf("\n Memory comparison:\n");
printf(" float32: %lu bytes\n",
M * K * sizeof(float));
printf(" int8: %lu bytes + %d scales (%lu "
"B)\n",
M * K * sizeof(signed char), M,
M * sizeof(float));
printf(" Ratio: %.1fx smaller\n",
(float)(M * K * sizeof(float)) /
(M * K * sizeof(signed char)
+ M * sizeof(float)));
/* Measure it rather than assert it */
volatile float sink = 0;
clock_t t0, t1;
int r, reps = 200000;
t0 = clock();
for (r = 0; r < reps; r++)
for (i = 0; i < M; i++) {
float a = 0;
for (j = 0; j < K; j++)
a += W[i][j] * x[j];
sink += a;
}
t1 = clock();
double t_f32 = (double)(t1 - t0) / CLOCKS_PER_SEC;
t0 = clock();
for (r = 0; r < reps; r++)
for (i = 0; i < M; i++) {
int acc = 0;
for (j = 0; j < K; j++)
acc += (int)W_q[i][j] * (int)x_q[j];
sink += acc * w_scales[i] * x_scale;
}
t1 = clock();
double t_int8 = (double)(t1 - t0) / CLOCKS_PER_SEC;
printf("\n Timing over %d repetitions:\n", reps);
printf(" float32: %.3f s\n", t_f32);
printf(" int8: %.3f s\n", t_int8);
printf(" speedup: %.2fx\n", t_f32 / t_int8);
printf("\n Dequantization runs once per "
"output,\n");
printf(" not once per multiply, so the inner "
"loop\n");
printf(" stays in integer arithmetic.\n");
return 0;
}
Figure 35-4 reports accuracy, memory and measured execution time for both paths. On examining our results, the timing result is the one which should give you pause. Look at what happened here, int8 took 1.675 seconds against 1.539 for float32 and finished at 0.92 of the float32 speed. Scalar C cannot express the fused integer dot product instructions that make quantized inference fast in production. So essentially, this loop measures one scalar multiply against another with a sign extension attached (sound familiar?). Compilers vectorize the floating point path using registers that have been present on commodity hardware for decades, while the integer path needs newer instructions and a more cooperative loop shape. A 64 by 64 matrix also occupies sixteen kilobytes, and lives entirely in first level cache, so the bandwidth advantage that motivates quantization never comes into play. Treat any speed claim in this area as a measurement you owe yourself on the machine you actually intend to deploy on. Trust me it’s better to test yourself than rely on assumptions.
What I want to tell you before you get too discouraged is that accuracy holds up well under quantization, even though at first glance and based on everything we have been saying it doesn’t feel like it. Since a maximum absolute error of 0.009734 lands against outputs which reach roughly 1.30 in magnitude, if measured against the largest output that amounts to 0.75 percent. That is the figure you keep in your head rather than an error taken relative to some arbitrarily chosen element. The root mean square error of 0.003891 is smaller again, which tells you the large errors are rare rather than typical across the sixty four outputs. The memory ratio settles at 3.8 rather than 4.0 because every row carries a scale of its own, which is the finest granularity this scheme allows and therefore the largest overhead it can incur. Coarser grouping would push the ratio closer to four at the cost of the resolution that per row scaling was introduced to provide.
35.6 Four Bit Quantization
I want us to draw specialized attention to 4-bit quantization, because it is without a doubt one you’ll encounter on real model deployments. Four bits yield sixteen levels. This is a “goldilocks” spot because it’s few enough that the packing becomes about as interesting as the quantization itself. Two values share a byte, one occupying the low nibble and one the high, and unpacking requires a sign extension the compiler will not perform on your behalf. The scale divides by seven rather than eight, keeping the mapping symmetric and leaving the most negative level unused exactly as the int8 scheme did. Everything else is identical to absmax, which is rather the point, since dropping from eight bits to four changes the error without changing the algorithm at all. Let that sink in for a bit.
The program we will write reports the four bit error alongside the int8 error on the same vector, so the comparison between them is measured rather than claimed.
/* 172_4bit.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N 16
/* Pack two 4-bit values into one byte */
static void quantize_4bit(const float *x,
unsigned char *packed,
float *scale, int n)
{
float absmax = 0;
int i;
for (i = 0; i < n; i++)
if (fabsf(x[i]) > absmax) absmax = fabsf(x[i]);
*scale = absmax / 7.0f; /* 4-bit signed: -8 to 7 */
if (*scale < 1e-10f) *scale = 1e-10f;
for (i = 0; i < n; i += 2) {
float a = x[i] / (*scale);
int v0 = (int)(a + (a >= 0 ? 0.5f : -0.5f));
float b = x[i+1] / (*scale);
int v1 = (int)(b + (b >= 0 ? 0.5f : -0.5f));
if (v0 > 7) v0 = 7;
if (v0 < -8) v0 = -8;
if (v1 > 7) v1 = 7;
if (v1 < -8) v1 = -8;
/* Pack: low nibble = v0, high nibble = v1 */
packed[i/2] = ((v1 & 0xF) << 4) | (v0 & 0xF);
}
}
static void dequantize_4bit(
const unsigned char *packed,
float scale, float *out,
int n)
{
int i;
for (i = 0; i < n; i += 2) {
int v0 = packed[i/2] & 0xF;
int v1 = (packed[i/2] >> 4) & 0xF;
/* Sign extend from 4 bits */
if (v0 & 0x8) v0 |= ~0xF;
if (v1 & 0x8) v1 |= ~0xF;
out[i] = v0 * scale;
out[i+1] = v1 * scale;
}
}
int main(void)
{
float x[N];
unsigned char packed[N/2];
float scale, x_hat[N];
int i;
srand(42);
for (i = 0; i < N; i++)
x[i] = (randf()*2-1) * 1.0f;
quantize_4bit(x, packed, &scale, N);
dequantize_4bit(packed, scale, x_hat, N);
printf("4-bit quantization (two values per "
"byte):\n\n");
printf(" Original Dequant Error Quant\n");
printf(" -------- -------- ------- -----\n");
for (i = 0; i < N; i++) {
int val;
if (i % 2 == 0) val = packed[i/2] & 0xF;
else val = (packed[i/2] >> 4) & 0xF;
if (val & 0x8) val |= ~0xF;
printf(" %+.4f %+.4f %.4f %3d\n",
x[i], x_hat[i],
fabsf(x[i] - x_hat[i]), val);
}
float rms = 0;
for (i = 0; i < N; i++)
rms += (x[i] - x_hat[i]) * (x[i] - x_hat[i]);
rms = sqrtf(rms / N);
printf("\n RMS error: %.4f\n", rms);
printf(" Memory: %d bytes (was %lu bytes)\n",
N/2 + 4, N * sizeof(float));
printf(" Compression: %.1fx\n",
(float)(N * sizeof(float)) / (N/2 + 4));
/* Same vector at int8, so the claim is measured */
float s8 = 0, rms8 = 0;
for (i = 0; i < N; i++)
if (fabsf(x[i]) > s8) s8 = fabsf(x[i]);
s8 /= 127.0f;
for (i = 0; i < N; i++) {
float v = x[i] / s8;
float d = (float)(int)(v
+ (v >= 0 ? 0.5f : -0.5f))
* s8;
rms8 += (x[i] - d) * (x[i] - d);
}
rms8 = sqrtf(rms8 / N);
printf(" int8 RMS on the same vector: "
"%.4f\n", rms8);
printf(" 4-bit error is %.0fx "
"larger for 2x less\n",
rms / rms8);
printf(" memory. Only 16 levels per value.\n");
printf(" Many LLMs run at 4-bit with little "
"loss\n");
printf(" because weight distributions are "
"nearly\n");
printf(" Gaussian and most values sit near "
"zero.\n");
return 0;
}
Figure 35-5 sets four bit quantization against int8 on the same vector. From our results, we see that root mean square error climbs to 0.0327 against 0.0027 for int8 on the same data. That’s a factor of twelve for one, further halving of the width. The worst individual value is off by 0.0688, which is seven percent of the largest weight anywhere in the vector. Two entries collapse to zero outright because both sit closer to the origin than half a scale step of 0.0707. Trained networks tolerate this because the absolute error stays bounded by half a step wherever a weight happens to sit, and the weights suffering the worst relative error are the small ones. Those small weights contribute correspondingly little to any output, which is the entire reason four bit inference remains usable at all.
Formats such as NF4 improve on the uniform grid by spacing the sixteen levels, according to a normal distribution instead of evenly. You see what it does is place more of them near zero where trained weights actually concentrate. Should you choose to, the exercises ask you to build one and compare it against the uniform scheme on Gaussian data. Compression lands at 5.3 here for the same reason absmax landed at 2.7, since a four byte scale amortized across only sixteen values remains substantial overhead. Real deployments quantize in groups of sixty four or one hundred and twenty eight, at which point the packed nibbles dominate and the ratio approaches the eight the format promises. The error, unlike the compression ratio, does not improve with group size and we can attribute it to simply the price of sixteen levels.
35.7 Key Takeaways
Quantization converts float weights to lower precision integers, reducing memory by a factor of four at eight bits and eight at four bits, while enabling integer arithmetic in the inner loop.
Absmax quantization scales the entire tensor by its maximum absolute value. It is simple, symmetric about zero, and highly sensitive to outliers.
The scale is fixed overhead. At eight values per scale the measured reduction is only 2.7x, and reaching the advertised ratio requires amortizing each scale across a hundred or more values.
Group quantization gives each block of weights an independent scale, which quarantines an outlier inside its own group rather than removing its effect. Clean groups improved by sixteen to twenty times here while the group holding the outlier improved not at all.
Quantized matrix multiply accumulates in int32 and dequantizes once per output element. Scalar C measured 0.92 of the float32 speed on a cache resident problem, because production speedups come from fused integer instructions and from bandwidth savings that a 64 by 64 matrix cannot exhibit.
Four bit quantization packs two weights per byte and carried twelve times the error of int8 on the same data, which trained networks tolerate because the largest relative errors fall on the smallest and least influential weights.
DeepSeek-V3 trains in FP8 rather than merely running inference there, using fine grained quantization with scales computed online. The relative loss error against BF16 stays under 0.25 percent.
35.8 Exercises
Quantize a vector with one extreme outlier such as 0.01, 0.02, 0.01 and 100.0 to int8 with absmax. What happens to the small values? Now try group quantization and report the per group error.
Implement NF4, the four bit normal float format. Instead of uniform spacing from negative eight to seven, use sixteen levels optimized for a Gaussian distribution. Compare the error against uniform int4 on Gaussian distributed weights.
Measure the speedup of int8 matmul against float32 matmul using matrices of 1024 by 1024, which will not fit in cache. Compare the result to the 64 by 64 figure this chapter reports and account for the difference.
Implement asymmetric quantization, which adds a zero point offset so that q equals round of x over scale plus zero_point. Determine when the extra offset earns its keep against the symmetric scheme.
Quantize the weights of the model from Chapter 26 to int8, run inference, and compare the output against float32 inference. How much does the prediction change?
DeepSeek-V3 uses E4M3 for all FP8 tensors rather than reserving E5M2 for the backward pass. Explain why fine grained quantization is what permits the higher precision mantissa format to be used everywhere.