Hidden Representations
Visualizing what hidden layers learn
7.1 What You Will Learn
In Chapter 2 we added a hidden layer and it solved XOR. In Chapter 6 we used 20 hidden neurons on a circle classification problem. So far we have been evaluating results, but we never looked at what those hidden neurons actually learned. In this chapter we open the “black box” and you will inspect hidden layer outputs, understand what each neuron detects, see how the network transforms input space into a space where classification is easy, and build tools to extract and compare representations. This chapter has no new math or concepts, it is about building intuition for what neural networks do internally.
7.2 The Idea
When an input passes through a hidden layer, it comes out the other side as a different set of numbers. Those new numbers are a representation of the original input, and the network learns to shape that representation so that whatever comes next, whether that is classification, prediction, or regression (regression is like prediction, but of a continuous number instead of a category, like classification will say this is Class A or B and regression will be like this is 3.7 or this 2.1), becomes as simple as possible. The easiest way to think about this is as a coordinate transform, you start with data that is tangled together in its original coordinates, impossible to separate with a straight line. The hidden layer rotates, stretches, and warps that space until the classes fall neatly apart. Once the data is rearranged like that, the output layer only has to draw a simple boundary through the transformed space to get the right answer.
We already saw this happen with XOR. The four input pairs, (0,0), (0,1), (1,0), (1,1), cannot be separated by a single line in their original 2D space. But after passing through the hidden layer, those same four points land in a new 2D space where they can. Let us go back to the XOR network and look carefully at what the hidden layer actually produces.
Figure 7-1 puts the two spaces side by side using the numbers the first program in this chapter prints. On the left are the four inputs where they started, with the two that should output 1 sitting on one diagonal and the two that should output 0 on the other, which is the arrangement no straight line can cut. On the right are the same four inputs plotted by their hidden activations h0 and h1 instead of by x1 and x2, and a single straight line now does the job.
Two details are worth pausing on. The hidden layer did not add a dimension, since both spaces are two dimensional, so the network did not solve XOR by giving itself more room to work in. It solved it by moving the points. And the two inputs that share an answer, (0,1) and (1,0), have been pushed to almost the same place, close enough that they overlap in the plot. The network has decided those two inputs are the same kind of thing, which is what a useful representation does. It throws away the distinction that does not matter and keeps the one that does.
7.3 XOR Hidden Layer Outputs
Let us train an XOR network and then print what the hidden neurons output for each input. We already did this briefly in Chapter 2, but now we will look more carefully. Setup the network, by this point you could do this in your sleep.
/* 039_XOR_Hidden.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float sigmoid(float z)
{
return 1.0f / (1.0f + expf(-z));
}
typedef struct { float wh[2][2],
bh[2];
float wo[2], bo;
}
Net;
static float forward(const Net *n,
const float x[2], float h[2])
{
int i, j;
for (i = 0; i < 2; i++) {
float z = n->bh[i];
for (j = 0; j < 2; j++) z += n->wh[i][j] * x[j];
h[i] = sigmoid(z);
}
{ float z = n->bo;
for (i = 0; i < 2; i++) z += n->wo[i] * h[i];
return sigmoid(z);
}
}
static void backward(Net *n, const float x[2],
const float h[2],
float y, float t, float lr)
{
int i, j;
float d_out = -2.0f * (t - y) * y * (1.0f - y);
float d_h[2];
for (i = 0; i < 2; i++)
d_h[i] = d_out * n->wo[i] * h[i]
* (1.0f - h[i]);
for (i = 0; i < 2; i++)
n->wo[i] -= lr * d_out * h[i];
n->bo -= lr * d_out;
for (i = 0; i < 2; i++) {
for (j = 0; j < 2; j++)
n->wh[i][j] -= lr * d_h[i] * x[j];
n->bh[i] -= lr * d_h[i];
}
}
int main(void)
{
float X[4][2] = { {0, 0}, {0, 1}, {1, 0}, {1, 1} };
float T[4] = { 0, 1, 1, 0 };
Net net;
int epoch, s, i, j;
srand(42);
for (i = 0; i < 2; i++) {
for (j = 0; j < 2; j++)
net.wh[i][j] = ((float)rand()/RAND_MAX)*2-1;
net.bh[i] = ((float)rand()/RAND_MAX)*2-1;
net.wo[i] = ((float)rand()/RAND_MAX)*2-1;
}
net.bo = ((float)rand()/RAND_MAX)*2-1;
for (epoch = 0; epoch < 10000; epoch++) {
for (s = 0; s < 4; s++) {
float h[2], y;
y = forward(&net, X[s], h);
backward(&net, X[s], h, y, T[s], 1.0f);
}
}
printf("XOR hidden layer representations:\n\n");
printf(" input h0 h1 output "
"target\n");
printf(" ------ ------ ------ ------ "
"------\n");
for (s = 0; s < 4; s++) {
float h[2], y;
y = forward(&net, X[s], h);
printf(" (%.0f, %.0f) %6.4f %6.4f %6.4f "
"%.0f\n",
X[s][0], X[s][1], h[0], h[1], y, T[s]);
}
printf("\nNow look at just h0 and h1 as a 2D "
"space:\n");
printf(" The four input pairs map to four points "
"in hidden space.\n");
printf(" In hidden space, class 1 (XOR=1) is "
"linearly separable\n");
printf(" from class 0 (XOR=0).\n");
return 0;
}
Figure 7-2 places the four XOR inputs as four points in hidden space. Study the h0 and h1 columns. The four inputs map to four points in 2D hidden space. The key thing to notice: the two XOR=1 cases land in a different region of hidden space than the two XOR=0 cases. The hidden layer has rearranged the points so that the output neuron can draw a single line between them. Each hidden neuron has learned to detect a different feature of the input. The exact features depend on the random initialization, but the result is always the same: the transformed space is linearly separable.
7.4 Circle Classification Hidden Space
XOR only has four points, so there is not much to look at. A more interesting case is the circle classification problem from Chapter 6, where points inside a circle belong to one class and points outside belong to another. We will train a network on that problem and then print the hidden layer outputs for a grid of inputs. This lets us see how the hidden layer takes the original 2D space and warps it into something new, stretching and bending the coordinates until the two classes become easy to separate with a straight line.
/* 040_Circle_Hidden.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float sigmoid(float z)
{
return 1.0f / (1.0f + expf(-z));
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N_HID 4 /* small enough to inspect */
#define N_PARAMS (2*N_HID + N_HID + N_HID + 1)
static float forward(const float *p, const float x[2],
float h[N_HID])
{
int i, j;
for (i = 0; i < N_HID; i++) {
float z = p[2*N_HID + i];
for (j = 0; j < 2; j++) z += p[i*2+j] * x[j];
h[i] = sigmoid(z);
}
{ int bw = 3*N_HID; float z = p[bw+N_HID];
for (i = 0; i < N_HID; i++) z += p[bw+i] * h[i];
return sigmoid(z);
}
}
static void backward(const float *p, const float x[2],
const float h[N_HID],
float y, float t, float *g)
{
int i, j, bh = 2*N_HID, bw = 3*N_HID;
float d_out = -2.0f * (t - y) * y * (1.0f - y);
for (i = 0; i < N_PARAMS; i++) g[i] = 0;
for (i = 0; i < N_HID; i++) g[bw+i] = d_out * h[i];
g[bw+N_HID] = d_out;
for (i = 0; i < N_HID; i++) {
float dh = d_out * p[bw+i] * h[i]
* (1.0f - h[i]);
for (j = 0; j < 2; j++) g[i*2+j] = dh * x[j];
g[bh+i] = dh;
}
}
typedef struct { float *m, *v; float b1, b2, eps, lr,
b1t, b2t;
int n;
}
Adam;
static Adam adam_create(int n, float lr) {
Adam o;
o.m = (float*)calloc(n, sizeof(float));
o.v = (float*)calloc(n, sizeof(float));
o.b1 = 0.9f;
o.b2 = 0.999f;
o.eps = 1e-8f;
o.lr = lr;
o.b1t = 1;
o.b2t = 1;
o.n = n;
return o;
}
static void adam_update(Adam *o, float *p,
const float *g) {
int i;
o->b1t *= o->b1;
o->b2t *= o->b2;
for(i = 0;i<o->n;i++) {
o->m[i] = o->b1*o->m[i]+(1-o->b1)*g[i];
o->v[i] = o->b2*o->v[i]+(1-o->b2)*g[i]*g[i];
float mh = o->m[i]/(1-o->b1t),
vh = o->v[i]/(1-o->b2t);
p[i] -= o->lr*mh/(sqrtf(vh)+o->eps);
}
}
static void adam_free(Adam *o)
{
free(o->m);
free(o->v);
}
typedef struct { float x[2]; float t; } Sample;
int main(void)
{
Sample data[200];
float params[N_PARAMS], grads[N_PARAMS],
ga[N_PARAMS];
Adam opt;
int epoch, s, i;
srand(42);
/* Generate circle data */
for (i = 0; i < 200; i++) {
data[i].x[0] = randf() * 4 - 2;
data[i].x[1] = randf() * 4 - 2;
float dist = data[i].x[0]*data[i].x[0]
+ data[i].x[1]*data[i].x[1];
data[i].t = (dist < 1.44f) ? 1.0f : 0.0f;
/* no noise for clarity */
}
for (i = 0; i < N_PARAMS; i++)
params[i] = randf() * 0.4f - 0.2f;
opt = adam_create(N_PARAMS, 0.01f);
/* Train */
for (epoch = 0; epoch < 1000; epoch++) {
for (i = 0; i < N_PARAMS; i++) ga[i] = 0;
for (s = 0; s < 200; s++) {
float h[N_HID], y;
y = forward(params, data[s].x, h);
backward(params, data[s].x, h, y,
data[s].t, grads);
for (i = 0; i < N_PARAMS; i++)
ga[i] += grads[i];
}
for (i = 0; i < N_PARAMS; i++) ga[i] /= 200.0f;
adam_update(&opt, params, ga);
}
/* Inspect: what does each hidden neuron respond
to? */
printf("Hidden neuron weights (what each neuron "
"detects):\n\n");
for (i = 0; i < N_HID; i++) {
printf(" Neuron %d: w0=%+.3f w1=%+.3f "
"bias=%+.3f\n",
i, params[i*2], params[i*2+1],
params[2*N_HID+i]);
}
/* Sample a few points and show their hidden
representations */
printf("\nHidden representations for selected "
"points:\n\n");
printf(" x0 x1 dist ");
for (i = 0; i < N_HID; i++) printf(" h%d ", i);
printf(" output class\n");
float test_points[][2] = {
{ 0.0f, 0.0f}, /* center */
{ 0.5f, 0.5f}, /* inside */
{-0.8f, 0.3f}, /* inside */
{ 1.0f, 0.0f}, /* on boundary */
{ 1.5f, 0.0f}, /* outside */
{ 0.0f, 1.5f}, /* outside */
{-1.5f, -1.5f}, /* far outside */
};
int n_test = 7;
for (s = 0; s < n_test; s++) {
float h[N_HID], y;
float dist = test_points[s][0]*test_points[s][0]
+ test_points[s][1]
*test_points[s][1];
y = forward(params, test_points[s], h);
printf(" %+5.1f %+5.1f %5.2f",
test_points[s][0], test_points[s][1], dist);
for (i = 0; i < N_HID; i++)
printf(" %5.3f", h[i]);
printf(" %5.3f %s\n", y,
dist < 1.44f ? "IN" : "OUT");
}
adam_free(&opt);
return 0;
}
Figure 7-3 lists four hidden neurons, their weights, and what each one outputs across the circle. Each hidden neuron has two weights and a bias, and those three numbers define a line in 2D space. On one side of that line the neuron fires strongly, on the other side it stays quiet. With four neurons you get four lines, and together those lines carve the input plane into regions. The network learns to position those lines so that points inside the circle produce one pattern of activations across the four neurons, and points outside produce a different pattern. The output neuron then just has to recognize which pattern it is looking at. You can see this in the table. The four IN points all share a similar signature: h0 through h3 are all low, near zero, and the output is high. The moment you cross the boundary, something changes. At (1.5, 0.0) neuron h0 jumps to 0.958 while the output drops to 0.078. At (0.0, 1.5) neuron h2 jumps to 0.594 and at (-1.5, −1.5) neuron h3 shoots up to 0.983. Each neuron is guarding a different side of the circle. When any of them fires, it is telling the output layer “this point is outside my boundary,” and the output responds by dropping toward zero.
The original question the network had to answer was “is this point’s distance from the center less than 1.2?” That is a nonlinear question, and no single neuron can answer it. But four neurons, each watching a different edge of the circle, can collectively encode enough information that the output layer only has to ask a linear question: “are all the guard neurons quiet?” If yes, the point is inside. If any of them are firing, it is outside.
7.5 Watching Representations Form
The most illuminating thing is to watch the hidden representations change during training. Early on, the hidden outputs are random. As training proceeds, the representations reorganize until the classes become separable.
/* 041_Watching_Representations_Form.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float sigmoid(float z)
{
return 1.0f / (1.0f + expf(-z));
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N_HID 2 /* 2 hidden so we can think in 2D */
#define N_PARAMS (2*N_HID + N_HID + N_HID + 1)
static float forward(const float *p, const float x[2],
float h[N_HID])
{
int i, j;
for (i = 0; i < N_HID; i++) {
float z = p[2*N_HID + i];
for (j = 0; j < 2; j++) z += p[i*2+j] * x[j];
h[i] = sigmoid(z);
}
{ int bw = 3*N_HID; float z = p[bw+N_HID];
for (i = 0; i < N_HID; i++) z += p[bw+i] * h[i];
return sigmoid(z);
}
}
static void backward(const float *p, const float x[2],
const float h[N_HID],
float y, float t, float *g)
{
int i, j, bh = 2*N_HID, bw = 3*N_HID;
float d_out = -2.0f * (t - y) * y * (1.0f - y);
for (i = 0; i < N_PARAMS; i++) g[i] = 0;
for (i = 0; i < N_HID; i++) g[bw+i] = d_out * h[i];
g[bw+N_HID] = d_out;
for (i = 0; i < N_HID; i++) {
float dh = d_out * p[bw+i] * h[i]
* (1.0f - h[i]);
for (j = 0; j < 2; j++) g[i*2+j] = dh * x[j];
g[bh+i] = dh;
}
}
typedef struct { float *m, *v; float b1, b2, eps, lr,
b1t, b2t;
int n;
}
Adam;
static Adam adam_create(int n, float lr) {
Adam o;
o.m = (float*)calloc(n, sizeof(float));
o.v = (float*)calloc(n, sizeof(float));
o.b1 = 0.9f;
o.b2 = 0.999f;
o.eps = 1e-8f;
o.lr = lr;
o.b1t = 1;
o.b2t = 1;
o.n = n;
return o;
}
static void adam_update(Adam *o, float *p,
const float *g) {
int i;
o->b1t *= o->b1;
o->b2t *= o->b2;
for(i = 0;i<o->n;i++) {
o->m[i] = o->b1*o->m[i]+(1-o->b1)*g[i];
o->v[i] = o->b2*o->v[i]+(1-o->b2)*g[i]*g[i];
float mh = o->m[i]/(1-o->b1t),
vh = o->v[i]/(1-o->b2t);
p[i] -= o->lr*mh/(sqrtf(vh)+o->eps);
}
}
static void adam_free(Adam *o)
{
free(o->m);
free(o->v);
}
int main(void)
{
/* XOR so we can use 2 hidden neurons */
float X[4][2] = { {0, 0}, {0, 1}, {1, 0}, {1, 1} };
float T[4] = { 0, 1, 1, 0 };
float params[N_PARAMS], grads[N_PARAMS],
ga[N_PARAMS];
Adam opt;
int epoch, s, i;
int snapshots[] = { 0, 10, 50, 200, 1000, 5000 };
int n_snaps = 6;
int snap_idx = 0;
srand(42);
for (i = 0; i < N_PARAMS; i++)
params[i] = randf() * 2 - 1;
opt = adam_create(N_PARAMS, 0.01f);
printf("How hidden representations form during "
"training (2 hidden neurons):\n\n");
for (epoch = 0; epoch <= 5000; epoch++) {
/* Print snapshot */
if (snap_idx < n_snaps
&& epoch == snapshots[snap_idx]) {
printf("Epoch %d:\n", epoch);
printf(" input h0 h1 "
"output\n");
for (s = 0; s < 4; s++) {
float h[N_HID], y;
y = forward(params, X[s], h);
printf(" (%.0f, %.0f) %6.4f %6.4f "
"%6.4f (target %.0f)\n",
X[s][0], X[s][1], h[0],
h[1], y, T[s]);
}
printf("\n");
snap_idx++;
}
/* Train one epoch */
for (i = 0; i < N_PARAMS; i++) ga[i] = 0;
for (s = 0; s < 4; s++) {
float h[N_HID], y;
y = forward(params, X[s], h);
backward(params, X[s], h, y, T[s], grads);
for (i = 0; i < N_PARAMS; i++)
ga[i] += grads[i];
}
for (i = 0; i < N_PARAMS; i++) ga[i] /= 4.0f;
adam_update(&opt, params, ga);
}
adam_free(&opt);
return 0;
}
Figure 7-4 samples the hidden representation at four points in training, and it goes from jumbled to separated. At epoch 0 the hidden outputs are essentially random. The four inputs produce h0 and h1 values that are jumbled together with no clear structure, and the output sits around 0.5 for everything. The network knows nothing yet.
By epoch 200, things are starting to shift. The two class-0 inputs, (0,0) and (1,1), are drifting apart from the two class-1 inputs in hidden space, but the separation is still messy. The outputs are in the 0.2 to 0.5 range, showing the network has a rough sense of the answer but is not confident.
By epoch 1000 the pattern is clear. The two XOR=1 cases, (0,1) and (1,0), have nearly identical hidden representations: both have h0 near 0.02 and h1 near 0.07. Meanwhile the two XOR=0 cases have pushed to opposite corners, with (0,0) at high h0, low h1 and (1,1) at low h0, high h1. The network has learned that “both neurons low” means the answer is 1, and “either neuron high” means the answer is 0. The outputs are above 0.93 for the correct class-1 inputs and below 0.07 for class 0.
By epoch 5000 the separation has sharpened further. The h0 and h1 values for the XOR=1 cases are pushed even closer to zero, while the XOR=0 cases are pushed harder toward their respective corners. The outputs are now above 0.98 for class 1 and below 0.01 for class 0. The network found a 2D arrangement of the four points where a single line can separate the two classes, and then kept pushing those points further apart to increase its confidence.
This reorganization is learning, the network is not simply memorizing input and output pairs. It is discovering a coordinate system in which the problem is easy.
7.6 What Individual Neurons Detect
Each hidden neuron computes sigmoid(w0x0 + w1x1 + b), and the weights and bias define a line in 2D input space where w0x0 + w1x1 + b = 0. On one side of that line the neuron outputs a value near 1, and on the other side it outputs near 0. The weight vector (w0, w1) points perpendicular to the line, toward the side where the neuron activates strongly. The output weight connecting each hidden neuron to the output layer tells us how that neuron’s vote gets used. A positive output weight means the neuron is voting “inside the circle” when it fires, and a negative output weight means it is voting “outside.” The angle of the weight vector tells us which direction in the input plane that neuron is watching.
We will train on the circle problem again and then print each neuron’s decision line equation, its angle, and whether it votes for or against the inside class.
/* 042_Neuron_Lines.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float sigmoid(float z)
{
return 1.0f / (1.0f + expf(-z));
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N_HID 4
#define N_PARAMS (2*N_HID + N_HID + N_HID + 1)
static float forward(const float *p, const float x[2],
float h[N_HID])
{
int i, j;
for (i = 0; i < N_HID; i++) {
float z = p[2*N_HID + i];
for (j = 0; j < 2; j++) z += p[i*2+j] * x[j];
h[i] = sigmoid(z);
}
{ int bw = 3*N_HID; float z = p[bw+N_HID];
for (i = 0; i < N_HID; i++) z += p[bw+i] * h[i];
return sigmoid(z);
}
}
static void backward(const float *p, const float x[2],
const float h[N_HID],
float y, float t, float *g)
{
int i, j, bh = 2*N_HID, bw = 3*N_HID;
float d_out = -2.0f * (t - y) * y * (1.0f - y);
for (i = 0; i < N_PARAMS; i++) g[i] = 0;
for (i = 0; i < N_HID; i++) g[bw+i] = d_out * h[i];
g[bw+N_HID] = d_out;
for (i = 0; i < N_HID; i++) {
float dh = d_out * p[bw+i] * h[i]
* (1.0f - h[i]);
for (j = 0; j < 2; j++) g[i*2+j] = dh * x[j];
g[bh+i] = dh;
}
}
typedef struct { float *m, *v; float b1, b2, eps, lr,
b1t, b2t;
int n;
}
Adam;
static Adam adam_create(int n, float lr) {
Adam o;
o.m = (float*)calloc(n, sizeof(float));
o.v = (float*)calloc(n, sizeof(float));
o.b1 = 0.9f;
o.b2 = 0.999f;
o.eps = 1e-8f;
o.lr = lr;
o.b1t = 1;
o.b2t = 1;
o.n = n;
return o;
}
static void adam_update(Adam *o, float *p,
const float *g) {
int i;
o->b1t *= o->b1;
o->b2t *= o->b2;
for(i = 0;i<o->n;i++) {
o->m[i] = o->b1*o->m[i]+(1-o->b1)*g[i];
o->v[i] = o->b2*o->v[i]+(1-o->b2)*g[i]*g[i];
float mh = o->m[i]/(1-o->b1t),
vh = o->v[i]/(1-o->b2t);
p[i] -= o->lr*mh/(sqrtf(vh)+o->eps);
}
}
static void adam_free(Adam *o)
{
free(o->m);
free(o->v);
}
typedef struct { float x[2]; float t; } Sample;
int main(void)
{
Sample data[200];
float params[N_PARAMS], grads[N_PARAMS],
ga[N_PARAMS];
Adam opt;
int epoch, s, i;
srand(42);
for (i = 0; i < 200; i++) {
data[i].x[0] = randf() * 4 - 2;
data[i].x[1] = randf() * 4 - 2;
float dist = data[i].x[0]*data[i].x[0]
+ data[i].x[1]*data[i].x[1];
data[i].t = (dist < 1.44f) ? 1.0f : 0.0f;
}
for (i = 0; i < N_PARAMS; i++)
params[i] = randf() * 0.4f - 0.2f;
opt = adam_create(N_PARAMS, 0.01f);
for (epoch = 0; epoch < 2000; epoch++) {
for (i = 0; i < N_PARAMS; i++) ga[i] = 0;
for (s = 0; s < 200; s++) {
float h[N_HID], y;
y = forward(params, data[s].x, h);
backward(params, data[s].x, h, y,
data[s].t, grads);
for (i = 0; i < N_PARAMS; i++)
ga[i] += grads[i];
}
for (i = 0; i < N_PARAMS; i++) ga[i] /= 200.0f;
adam_update(&opt, params, ga);
}
printf("Each hidden neuron defines a decision "
"line:\n");
printf(" w0*x0 + w1*x1 + b = 0\n\n");
for (i = 0; i < N_HID; i++) {
float w0 = params[i*2];
float w1 = params[i*2+1];
float b = params[2*N_HID+i];
float angle = atan2f(w1,
w0) * 180.0f / 3.14159f;
printf(" Neuron %d: %+.3f*x0 + %+.3f*x1 + "
"%+.3f = 0\n",
i, w0, w1, b);
printf(" direction: "
"%.0f degrees", angle);
/* Output weight tells us if this neuron
votes for or against */
float ow = params[3*N_HID + i];
printf(" output weight: %+.3f (%s)\n",
ow, ow > 0 ? "votes IN" : "votes OUT");
}
/* Accuracy check */
int correct = 0;
for (s = 0; s < 200; s++) {
float h[N_HID], y;
y = forward(params, data[s].x, h);
int pred = y > 0.5f ? 1 : 0;
int actual = data[s].t > 0.5f ? 1 : 0;
if (pred == actual) correct++;
}
printf("\nAccuracy: %d/200 (%.1f%%)\n", correct,
correct / 2.0f);
adam_free(&opt);
return 0;
}
Figure 7-5 draws each hidden neuron as a line and marks the direction it faces. The four neurons have positioned their decision lines at 32, −44, 141, and −118 degrees, which means they are spread roughly 90 degrees apart around the circle. Each one is watching a different quadrant of the input plane. All four have negative output weights, so they are all voting “outside” when they fire. The output neuron has learned that when none of the hidden neurons are firing strongly, the point must be inside the circle, and when any of them activate, the point is outside. This is the same “guard neuron” pattern we saw earlier.
Since each neuron contributes one straight decision line, four neurons can only approximate the circle as a rough polygon. You are fitting a curved boundary with flat edges. More hidden neurons would give more edges and a tighter fit to the actual circle, but that also means more parameters, which is exactly why larger networks are more prone to the overfitting we covered in Chapter 6. There is always a tradeoff between how complex a boundary the network can represent and how well it generalizes to new data. The network hit 200/200 accuracy on the training set, which means even a crude four-sided approximation is enough to correctly classify every point in this dataset. But remember, this is the training set. Whether those four lines generalize to new points depends on whether the polygon they form has enough margin around the true circle boundary.
7.7 Representation Similarity
A powerful way to understand representations is to ask: which inputs does the network consider similar? Two inputs with similar hidden layer outputs will get similar predictions. Let us compute the distance between hidden representations of different inputs.
/* 043_Representation_Similarity.c */
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
static float sigmoid(float z)
{
return 1.0f / (1.0f + expf(-z));
}
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
#define N_HID 8
#define N_PARAMS (2*N_HID + N_HID + N_HID + 1)
static float forward(const float *p, const float x[2],
float h[N_HID])
{
int i, j;
for (i = 0; i < N_HID; i++) {
float z = p[2*N_HID + i];
for (j = 0; j < 2; j++) z += p[i*2+j] * x[j];
h[i] = sigmoid(z);
}
{ int bw = 3*N_HID; float z = p[bw+N_HID];
for (i = 0; i < N_HID; i++) z += p[bw+i] * h[i];
return sigmoid(z);
}
}
static void backward(const float *p, const float x[2],
const float h[N_HID],
float y, float t, float *g)
{
int i, j, bh = 2*N_HID, bw = 3*N_HID;
float d_out = -2.0f * (t - y) * y * (1.0f - y);
for (i = 0; i < N_PARAMS; i++) g[i] = 0;
for (i = 0; i < N_HID; i++) g[bw+i] = d_out * h[i];
g[bw+N_HID] = d_out;
for (i = 0; i < N_HID; i++) {
float dh = d_out * p[bw+i] * h[i]
* (1.0f - h[i]);
for (j = 0; j < 2; j++) g[i*2+j] = dh * x[j];
g[bh+i] = dh;
}
}
typedef struct { float *m, *v; float b1, b2, eps, lr,
b1t, b2t;
int n;
}
Adam;
static Adam adam_create(int n, float lr) {
Adam o;
o.m = (float*)calloc(n, sizeof(float));
o.v = (float*)calloc(n, sizeof(float));
o.b1 = 0.9f;
o.b2 = 0.999f;
o.eps = 1e-8f;
o.lr = lr;
o.b1t = 1;
o.b2t = 1;
o.n = n;
return o;
}
static void adam_update(Adam *o, float *p,
const float *g) {
int i;
o->b1t *= o->b1;
o->b2t *= o->b2;
for(i = 0;i<o->n;i++) {
o->m[i] = o->b1*o->m[i]+(1-o->b1)*g[i];
o->v[i] = o->b2*o->v[i]+(1-o->b2)*g[i]*g[i];
float mh = o->m[i]/(1-o->b1t),
vh = o->v[i]/(1-o->b2t);
p[i] -= o->lr*mh/(sqrtf(vh)+o->eps);
}
}
static void adam_free(Adam *o)
{
free(o->m);
free(o->v);
}
static float dist(const float *a, const float *b, int n)
{
float sum = 0;
int i;
for (i = 0; i < n; i++) {
float d = a[i] - b[i];
sum += d * d;
}
return sqrtf(sum);
}
int main(void)
{
float data_x[200][2], data_t[200];
float params[N_PARAMS], grads[N_PARAMS],
ga[N_PARAMS];
Adam opt;
int epoch, s, i;
srand(42);
for (i = 0; i < 200; i++) {
data_x[i][0] = randf() * 4 - 2;
data_x[i][1] = randf() * 4 - 2;
float d = data_x[i][0]*data_x[i][0]
+ data_x[i][1]*data_x[i][1];
data_t[i] = (d < 1.44f) ? 1.0f : 0.0f;
}
for (i = 0; i < N_PARAMS; i++)
params[i] = randf() * 0.4f - 0.2f;
opt = adam_create(N_PARAMS, 0.01f);
for (epoch = 0; epoch < 2000; epoch++) {
for (i = 0; i < N_PARAMS; i++) ga[i] = 0;
for (s = 0; s < 200; s++) {
float h[N_HID], y;
y = forward(params, data_x[s], h);
backward(params, data_x[s], h, y,
data_t[s], grads);
for (i = 0; i < N_PARAMS; i++)
ga[i] += grads[i];
}
for (i = 0; i < N_PARAMS; i++) ga[i] /= 200.0f;
adam_update(&opt, params, ga);
}
/* Pick a probe point and find nearest/farthest
in hidden space */
float probe[2] = { 0.3f, 0.4f };
float h_probe[N_HID];
forward(params, probe, h_probe);
printf("Probe point: (%+.1f, %+.1f)\n",
probe[0], probe[1]);
printf(" Input distance vs Hidden distance:\n\n");
printf(" point input_dist hidden_dist "
"same_class?\n");
float test_pts[][2] = {
{ 0.4f, 0.5f}, /* nearby, same class */
{-0.3f, -0.4f}, /* opposite side, same class */
/* nearby-ish, different class */
{ 1.8f, 0.0f},
{-1.8f, -1.8f}, /* far, different class */
{ 0.0f, 0.0f}, /* center, same class */
};
int n_test = 5;
for (s = 0; s < n_test; s++) {
float h_test[N_HID];
forward(params, test_pts[s], h_test);
float id = dist(probe, test_pts[s], 2);
float hd = dist(h_probe, h_test, N_HID);
float probe_dist = probe[0]*probe[0]
+ probe[1]*probe[1];
float test_dist = test_pts[s][0]
*test_pts[s][0] + test_pts[s][1]
*test_pts[s][1];
int same = ((probe_dist < 1.44f)
== (test_dist < 1.44f));
printf(" (%+4.1f,%+4.1f) %8.3f %8.3f "
"%s\n",
test_pts[s][0], test_pts[s][1], id, hd,
same ? "YES" : "NO");
}
printf("\nPoints in the same class should have "
"small hidden distance,\n");
printf("even if their input distance is large. The "
"network has learned\n");
printf("that (-0.3,-0.4) and (0.3,0.4) are 'the "
"same kind of thing.'\n");
adam_free(&opt);
return 0;
}
Figure 7-6 plots input distance against hidden distance for points around one probe. The key insight here is that input distance and hidden distance are different. Two points can be far apart in input space (opposite sides of the circle) but close in hidden space (both clearly inside). The network has learned that what matters is not position but relationship to the boundary. This is what representation learning means, that the network discovers the relevant features of the data and discards the irrelevant ones.
7.8 Why This Matters
Everything we build from here on comes back to representations. Convolutional networks learn representations of images where edges, textures, and shapes are explicitly encoded in the hidden layers. Recurrent networks learn representations of sequences where temporal patterns, the relationship between what came before and what comes next, get captured in a hidden state that evolves over time. Transformers learn representations of tokens where the meaning of a word shifts depending on the words around it. In every case, the quality of the hidden representation is what determines whether the network actually works. You can have millions of parameters and train for days, but if the network fails to find a useful way to represent the input internally, it will not solve the problem. On the other hand, a network that learns rich and informative representations can generalize well even when the training data is limited, because it has found the underlying structure rather than memorizing surface details.
When researchers say a model “understands” something, what they really mean is that its hidden representations have captured the relevant structure of the data. The model does not understand the way a person does, but the information is organized internally in a way that makes the right answers accessible. When we look inside a transformer later in this book, we will see exactly this principle at work, just operating over millions of parameters instead of four hidden neurons on a circle problem.
7.9 Key Takeaways
Hidden layers transform inputs into new representations. The network chooses representations that make the downstream task easy.
Each hidden neuron with sigmoid activation defines a decision line (or hyperplane in higher dimensions). The neuron activates on one side and deactivates on the other.
Multiple neurons together carve the input space into regions. Points in the same class end up in similar regions of hidden space, even if they are far apart in input space.
Representations form gradually during training. Early in training they are random. Backpropagation reorganizes them until classes are separable.
Representation similarity (distance in hidden space) reflects task-relevant similarity, not geometric similarity. The network learns what matters and ignores what doesn’t.
This principle scales from our 4-neuron circle classifier to billion-parameter language models. The difference is scale, not kind.
7.10 Exercises
Train an XOR network with 3 hidden neurons instead of 2. Print the hidden representations. Does the third neuron learn something different from the first two, or is it redundant?
Train the circle classifier with 2, 4, 8, and 16 hidden neurons. For each, count the number of correct classifications on 200 points. How many neurons does the circle need?
Compute the average hidden distance between same-class pairs and different-class pairs. The ratio of these two numbers tells you how well the representations separate the classes.
Save the hidden representations at epoch 0, 100, 500, and 2000 to a CSV file. If you have a plotting tool, plot h0 vs h1 colored by class at each snapshot to see the representations separate over time.
Train two networks with different random seeds on the same data. Compare their hidden representations for the same inputs. Are they the same? (They won’t be, but the structure should be equivalent.)
Add a second hidden layer of 2 neurons between the first hidden layer and the output. Print the representations at both layers. How does the second layer further transform the representation from the first?