I Trained GPT on a RISC-V MCU with No FPU in 37KB of Memory

So before the machine learning aficionados tell me how I'm "technically" wrong let me explain the architecture.

The model is a complete decoder only transformer with backpropagation and Adam optimiser, compiled for a Baochip-1x, which is a 350 MHz VexRISC-V SoC implementing RV32IMAC. There is no F extension in that string, which means the chip has no floating point unit, which means every single multiply and add inside that training loop is a function call into a software emulation routine. The whole thing, backward pass and optimiser and all, compiles to 37 KB of code, and it trains completely on chip from random weights in a little over an hour. So yes you can do this, and yes you don't need an FPU. 

A note on memory before anybody starts, because the computer architecture people will get to this before the machine learning people do. The 37 KB is code, and it lives in ReRAM, because the Baochip has no flash at all. It carries 4 MB of ReRAM, which is known as resistive RAM, which is non volatile and byte addressable and executes in place, along with 2 MB of regular SRAM. Whether ReRAM counts as memory or as storage is a question for industry guys who get to say what is defined as what, and I have opinions about that which belong in a different article, so for my purpose I'll say its memory cause that's what it is. The working set for training is a separate number and a much bigger one, about 1 MB of that SRAM, and there is a table further down that accounts for every byte of it.

Some background on where this came from

I have been writing a book (/ building a project?) for nearing on a year now with my very limited time, on building AI from scratch in C. Not using a framework or calling into a library, but writing everything by hand so that I, and then you who happens to read my work, understands what every line is doing inside the AI you eventually build for the capstone.

I know about llama2.c and llm.c and the recent surge into AI in C stuff, but I chose C just because I can think in C and my aim was understanding. I was more interested in understanding what makes models tick than deploy them, which I have enough experience doing.

Most "from scratch" AI tutorials always pull some library, and I got tired of that. Just give me nano or notepad and programs I can change and experiment with. And while I do have the context to remember snippets here and there, I don't like that approach, I like learning in chunks and then assembling everything.

Which is basically what I did. I wanted to start at the perceptron up to reading modern papers labs like DeepSeek publish and follow them, and before you go "ahh sales pitch" I figured a book like that will get pirated anyway and I want people (me included) to understand the internals of LLMs so when companies say "a text chat bot escaped to mars", we'll be able to separate the wheat from the chaff. So the book is free to read, it's not perfect, but I learnt a lot writing and working through it, and if you want to support the work you can pick up the PDF. Read it free online here: AI from Scratch in C. Okay, enough about that.

So, I'm doing final edits (a book ALWAYS has something wrong, no matter how many times you sweep it), then my Dabao boards arrived from Crowd Supply. I had earlier versions of the boards bunnie so graciously sent to me, and I also backed the campaign so those boards came. In case you don't know what the Dabao is, it's the evaluation board for the Baochip-1x, bunnie Huang's "mostly open" 22nm RISC-V chip, and I had already written a bare metal C SDK for it, so I had a board on my desk and a training loop in a text editor and the obvious thought followed. I had been troubleshooting some issues with the peripherals, and its already 10 pm on a Thursday night and I just posted my video on unboxing the boards, which you can see here, so, you know what popped into my head, I guess any engineer reading this knows. lol.

Yes. You guessed correct. 

The capstone was written for a desktop. It uses stdio and stdlib and libm and assumes a machine with an operating system and gigabytes of memory. Getting it onto a chip with none of those things is what this article is about, so for some context here are the model specs:

The essential architecture is all there, so we have token embedding and learned position embedding, RMSNorm, multi head attention with a causal mask, GELU, we have a feed forward block, residual connections and weight tying between the embedding and the output projection, and we also have softmax with cross entropy, and Adam. All the terms you see used when discussing modern LLMs. What it does not have is the stuff the frontier labs bolted on later, so no rotary embeddings, no grouped query attention, no SwiGLU, no KV cache. This is the GPT-2 shape with RMSNorm in it, which is the shape you need to understand before any of that later stuff means anything. Every component maps to a self contained program cause I was like if someone understands GELU today, but steps away and has to come back in a few weeks to learn multi head attention, or they wanna understand that gap they miss, they can turn to the thing they want to learn, read the chapter and understand whats going on. So anyway back to the implementation.

Why no FPU changes a lot

Every serious on device training project I could find runs on hardware with floating point support, even the on device training work on STM32F7 notes that the part is embedded with an FPU. The most similar project to this one, a char level transformer trained from scratch on an ESP32-S3, runs on a chip whose Xtensa core has a single precision FPU. The literature is explicit about why, though, gradient values can be very small and therefore require a high precision floating point unit to represent them, that not all microcontrollers even have floating point support, and that for the ones without, specific solutions need to be given.

I knew what the literature said, I've been deep in it for the past year or so, BUT here's my position though. I have chips with FPUs but the Baochip was on my desk so it seemed the more reasonable option. Even though I have a CH32H417 that has an FPU, its connected to another project in my lab. So the RV32IMAC Baochip that gives you integers, multiply, atomics and compressed instructions seemed more plausible than disconnecting my project with the CH32H417 and plugging it back in afterward. Even though I knew every float operation compiles to a call into libgcc and what the overhead might look like. 

Three things that stopped the build

1. In the Rovari workspace the drivers get pulled in through data/config.json rather than through the header include, so an empty peripherals array gives you undefined references to trng_init, rram_read and rram_write even though the headers are included. I ticked the boxes and that fixed it. 

2. Newlib's rand() touches the reentrancy structure, which drags in malloc, which pulls libnosys sbrk.o, which references an end symbol the linker script never defines. I replaced rand() with a nine line xorshift32 generator and defined _sbrk locally so sbrk.o never gets pulled in at all. That change turned out to matter far more than the linker error, and I will come back to it.

3. The mini_printf I wrote in my SDK handles bare %d, %c, %x and %s, which is entirely reasonable for a bare metal console. It does not handle %f, which I expected, and it also does not handle width or flag specifiers, which I did not. So %-9s printed literally, %3d printed literally, and worst of all %06d dropped its leading zeros, which would have turned a loss of 0.017977 into 0.17977 without any indication that anything was wrong. Go ahead, ask me how I know.

Luckily I'm a resourceful guy. Back in 2016 when I was still working heavily with 8-bit PIC microcontrollers I wrote an ftoa that walks the fractional part digit by digit, and you can see the original in EUSART.c from my PIC16 projects. The modded version is in the repo linked below with the full Baochip GPT. The two lines that add a half step before the split are load bearing, because without them each value *= 10.0f drifts slightly low in float32 and the last digit comes out one under, so 0.017977 prints as 0.017976 and 0.046150 prints as 0.046149. I only caught it because I was comparing against a desktop build digit by digit.

So then the next question was memory.

Where the memory goes

The model itself is 108 KB. Training costs nine and a half times that, because Adam carries two moment estimates alongside the gradients and the backward pass needs every intermediate activation held in memory. The single largest item in the cache is the post softmax attention matrix at 128 KB, and half of that is structurally zero, because everything above the diagonal is masked out and I am still paying full price for it. The whole working set comes to 1,046,140 bytes against the chip's 2 MB of SRAM, so it fits with a hair under half the SRAM to spare.

Then someone reading this will be like "you said 37 KB". Set that against the table for a moment. The 37 KB is code, not data. Everything that implements this, attention and the causal mask and the full backward pass and Adam and the gradient checker, along with the SDK drivers and libm and the entire software floating point runtime that makes any of it possible on this core, comes to 37 KB. The parameters alone are 108 KB, so the instructions that perform backpropagation take a third of the space of the numbers they operate on, and about an eleventh of the activation cache they write into. Training is expensive in memory and remarkably cheap in code, and I myself wasn't fully aware of how lopsided that ratio is until I started writing stuff by hand.

So then we have to start thinking about how long the model takes to train.

Where the time goes

A training step takes 3487 ms. At 350 MHz that is roughly 1.22 billion cycles for one gradient update on 27,680 parameters. I instrumented the transcendental calls to find out where it was going:

Which turns out not to be the answer. Even at a generous fifteen hundred cycles apiece those calls are about eight percent of the run. Building a tanh lookup table, which was my first instinct, would have bought almost nothing. The other ninety two percent is plain multiplication and addition. One training step is about 4.6 million float operations, roughly 1.37 million forward, 3.05 million backward, 220 thousand in Adam. That works out to around 265 cycles per float operation once you include call overhead and memory traffic.

The most important measurement of the whole project came from changing the optimisation level. On my desktop, -Os to -O2 gave a 1.63x speedup but on the Baochip it gave one percent, so 3522 ms became 3487 ms.

That gap is the entire story of soft float. On a machine with an FPU the compiler has instructions to schedule, reorder and vectorise. When every operation is an opaque call into a precompiled library the optimiser can only tidy the loop arithmetic around the calls, and the calls are the cost. There is no compiler flag that makes software floating point fast.

Verifying that it actually worked

This is the part I care about most, and it is the part most machine learning demonstrations skip or gloss over. A loss curve going down is not proof of correctness. A subtly wrong backward pass will still produce a falling loss, just a worse one, and you will never know that. So before the model trains a single step, the chip verifies its own gradients. It perturbs each parameter block along a random direction, measures the actual change in loss, and compares that against the analytic gradient projected onto the same direction. Averaging over a direction rather than checking single elements removes the float32 noise that makes element wise checks useless at this precision.

If any block disagrees the program halts and refuses to train, because a training run built on a wrong derivative is worse than no training run at all, then there is the result I did not expect. Because I replaced rand() with xorshift32 to solve a linker error, the desktop build and the Baochip build started from bit identical random weights, so I could compare them directly:

So we get forty six seconds per epoch, thirteen batches each, steady throughout. 4610 seconds total, which is an hour and seventeen minutes. Two baselines print at startup so the numbers mean something. Guessing uniformly across 28 symbols scores 3.332205, which is just ln(28), so you can tell from that number alone that the loss is a natural log cross entropy and not something I made up. Predicting from character frequency alone scores 2.557692. A model that finishes above the second one has learned nothing beyond which letters are common. The interesting window is epoch 30 to 40, where the loss falls from 0.316 to 0.031. That's tenfold in ten epochs, about eight minutes. Here is the generated text as it goes:

epoch 10    the ig small cathe the dog
epoch 20    the mat. sat on the mat. s
epoch 30    the mat. at. athe t. t sma
epoch 40    the mat. a cat ran. the bi
epoch 100   the mat. a cat ran. the bi

Notice that it does not improve monotonically. Epoch 20 reads better than epoch 30 even though epoch 30 has the lower loss, and I am leaving that in rather than picking nicer samples, because that is what twenty five characters out of a 27,680 parameter model looks like. The loss is the measurement. The sample is an anecdote.

Nothing changes after epoch 40. Everything past that is the loss getting smaller without the behaviour changing, which is the signature of a model that has finished learning and started memorising, and that is a real limitation of running 27,680 parameters against a 248 character corpus. Forty epochs would have taken thirty one minutes and produced the same text, so that is the honest number for how long this actually needs to run. The finished weights go into ReRAM, 108 KB written once after training and never inside the loop, because ReRAM has finite write endurance and Adam would destroy it. Pull the power, plug it back in, and the chip generates text immediately without retraining.

IEEE 754 requires correctly rounded results for addition, subtraction, multiplication, division and square root, and libgcc's software implementations of those are conformant, so the two machines do genuinely identical arithmetic despite one of them having no floating point hardware. That is why they start bit identical and stay bit identical for ten epochs.

Then they separate, and the reason is the part of the standard that is only a recommendation. expf, logf, tanhf and powf are not required to be correctly rounded, and no C library I know of complies with the recommendation that they should be, so newlib's versions and glibc's versions disagree in the last bit. I make 35,764 of those calls every step. Look back at the table and notice which function is the one that cannot be the culprit: sqrtf is the single transcendental up there that the standard actually mandates, and it is also the most called. One unit in the last place, the same kind visible in norm2_g above, gets picked up by Adam and amplified until the two runs are walking different paths down the same loss surface. They still land in the same place though, I guess that's just gradient descent being chaotic.

One more thing before somebody asks. Those ten bit identical epochs also tell you my desktop build never fused a multiply and an add into a single FMA anywhere in the forward pass, because if it had, the two builds would have separated at step one instead of epoch ten. The soft float path has no FMA to contract into, so the match is only possible if the hardware path didn't use one either.

The ReRAM in LLMs Bits

There is something almost funny about where those weights ended up.

I spent an hour and change running 4.6 million software emulated float operations per step on a core with no floating point hardware, then wrote the finished parameters into ReRAM, which is the same base technology a substantial body of research is trying to turn into a matrix multiplier.

A ReRAM crossbar stores a weight as a conductance rather than a number, so if you drive the rows with voltages representing your input vector, the current summing on each column is the dot product, computed by Ohm's law and Kirchhoff's current law instead of an instruction sequence. PRIME, from ISCA 2016, proposed configuring parts of a ReRAM main memory as neural network accelerators on exactly this basis. There is a whole line of work since, on bit sparsity in the crossbar, on reconfigurable in situ accelerators, on conductance aware quantization to deal with real cells only supporting a limited number of discrete conductance states.

The Baochip-1x does not do any of that. Its ReRAM is byte addressable non volatile memory on the normal memory bus. No sense amplifiers arranged for analog readout or any peripheral circuitry that turns a memory array into a multiplier. bunnie put ReRAM on this chip because it is a good replacement for flash on 22nm, not because he was building an accelerator, and nothing I did exploits the crossbar. But the weights of my transformer are sitting in resistive memory cells right now, as stored numbers, having been computed one soft float call at a time by a core a few millimetres away. The technology holding them is the technology people are trying to make compute them. Strange thing to notice while watching a serial console print a loss value every forty six seconds.

What is a Language Model Anyway? 

So what I've built here is by all definitions a language model. It is a real transformer performing real gradient descent, verified against numerical differentiation on the target hardware, on a processor that cannot multiply two floating point numbers without calling a function to do it. But it is also very small, trained on very little text, and trained past the point where learning is more memorisation than anything else. I'm telling you that rather than making you find it.

Which is the part that interests me, actually. It is a language model.

It models a distribution over tokens and predicts the next one, and that is the definition. There is no clause specifying how many parameters you need before the words apply to you. A go-kart is a vehicle. Nobody argues about that and nobody confuses it with an F1 car either, because the word is about what the thing is and not about how fast it goes. But I know that when I say "language model", the definition is going to get very precise very quickly, and I notice that the precision only ever arrives in one direction.

Anyway. the whole thing is on GitHub, and if you find something I got wrong I would rather hear it than not:

ArmstrongSubero/AI-from-Scratch-in-C: Neural networks from scratch in C, from one perceptron to a working GPT. Code for the book AI from Scratch in C.

Armstrong Subero is an embedded systems engineer and published author with Apress/Springer. He builds the Rovari RISC-V education platform from Trinidad and Tobago.