I Trained GPT on a RISC-V MCU with No FPU in 37KB of Memory

So before the machine learning aficionados tell me how I'm "technically" wrong let me explain the architecture.

The model is a complete decoder only transformer with backpropagation and Adam optimiser, compiled for a Baochip-1x, which is a 350 MHz VexRISC-V SoC implementing RV32IMAC. There is no F extension in that string, which means the chip has no floating point unit, which means every single multiply and add inside that training loop is a function call into a software emulation routine. So yes it is possible to train a language model in that little space, and yes it trains completely on chip from random weights in about an hour and yes you don't need an FPU. 

A note on memory, since somebody on the computer architecture side will raise it. The 37 KB figure is in the thing the Baochip technically doesn't have which is Flash. It carries 4 MB of ReRAM, which is known as resistive RAM, which is non volatile and byte addressable and executes in place, along with regular SRAM.  Whether that counts as memory or as storage is a question for industry guys who get to say what is defined as what, and I have opinions about that which belong in a different article, so for my purpose I'll say its memory cause that's what it is.

Some background on where this came from

I have been writing a book (/ building a project?) for nearing on a year now with my very limited time, on building AI from scratch in C. Not using a framework or calling into a library, but writing everything by hand so that I, and then you who happens to read my work, understands what every line is doing inside the AI you eventually build for the capstone.

I know about llama.c and llm.c and the recent surge into AI in C stuff, but I chose C just because I can think in C and my aim was understanding. I was more interested in understanding what makes models tick than deploy them, which I have enough experience doing.

Most "from scratch" AI tutorials always pull some library, and I got tired of that. Just give me nano or notepad and programs I can change and experiment with. And while I do have the context to remember snippets here and there, I don't like that approach, I like learning in chunks and then assembling everything.

Which is basically what I did. I wanted to start at the perceptron up to reading modern papers labs like DeepSeek publish and follow them, and before you go "ahh sales pitch" I figured a book like that will get pirated anyway and I want people (me included) to understand the internals of LLMs so when companies say "a text chat bot escaped to mars", we'll be able to separate the wheat from the chaff. So the book is free to read, it's not perfect, but I learnt a lot writing and working through it and if you want to support the work, you can pick up the PDF,  go on this page click read and you'll kinds understsnd why I even attempted to run a language model on a chip this small to begin with, click read free online, and well read free online AI from Scratch in C - Rovari , okay enough about that.

So, I'm doing final edits (a book ALWAYS has something wrong, no matter how many times you sweep it), then my Dabao boards arrived from Crowd Supply. I had earlier versions of the boards bunnie so graciously sent to me, and I also backed the campaign so those boards came. In case you don't know what the Dabao is, it's the evaluation board for the Baochip-1x, bunnie Huang's mostly open" 22nm RISC-V chip, and I had already written a bare metal C SDK for it, so I had a board on my desk and a training loop in a text editor and the obvious thought followed. I had been troubleshooting some issues with the peripherals, and its already 10 pm on a Thursday night and I just posted my video on unboxing the boards, which you can see here: here so, you know what popped into my head, I guess any engineer reading this knows. lol.

Yes. You guessed correct. 

The capstone was written for a desktop. It uses stdio and stdlib and libm and assumes a machine with an operating system and gigabytes of memory. Getting it onto a chip with none of those things is what this article is about, so for some context here are the model specs:

We have everything thats in a modern LLM so we have token embedding and learned position embedding, RMSNorm, multi head attention with a causal mask, GELU, we have a feed forward block, residual connections and weight tying between the embedding and the output projection, and we also have softmax with cross entropy, and Adam. All the terms you see used when discussing modern LLMs. It is the same architecture everybody is talking about, just very small. The keyword here though is  essential architecture. Every component maps to a self contained program cause I was like if someone understands GELU today, but steps away and has to come back in a few weeks to learn multi head attention, or they wanna understand that gap they miss, they can turn to the thing they want to learn, read the chapter and understand whats going on. So anyway back to the implementation.

Why no FPU changes a lot

Every serious on device training project I could find runs on hardware with floating point support, even the on device training work on STM32F7 notes that the part is embedded with an FPU. The most similar project to this one, a char level transformer trained from scratch on an ESP32-S3, runs on a chip whose Xtensa core has a single precision FPU. The literature is explicit about why, though, gradient values can be very small and therefore require a high precision floating point unit to represent them, that not all microcontrollers even have floating point support, and that for the ones without, specific solutions need to be given.

I knew what the literature said, I've been deep in it for the past year or so, BUT here's my position though. I have chips with FPUs but the Baochip was on my desk so it seemed the more reasonable option. Even though I have a CH32H417 that has an FPU, its connected to another project in my lab. So the RV32IMAC Baochip that gives you integers, multiply, atomics and compressed instructions seemed more plausible than disconnecting my project with the CH32H417 and plugging it back in afterward. Even though I knew every float operation compiles to a call into libgcc and what the overhead might look like. 

Three things that stopped the build

1. In the Rovari workspace the drivers get pulled in through data/config.json rather than through the header include, so an empty peripherals array gives you undefined references to trng_init, rram_read and rram_write even though the headers are included. I ticked the boxes and that fixed it. 

2. Newlib's rand() touches the reentrancy structure, which drags in malloc, which pulls libnosys sbrk.o, which references an end symbol the linker script never defines. I replaced rand() with a nine line xorshift32 generator and defined _sbrk locally so sbrk.o never gets pulled in at all. That change turned out to matter far more than the linker error, and I will come back to it.

3. The mini_printf I wrote in my SDK handles bare %d, %c, %x and %s, which is entirely reasonable for a bare metal console. It does not handle %f, which I expected, and it also does not handle width or flag specifiers, which I did not. So %-9s printed literally, %3d printed literally, and worst of all %06d dropped its leading zeros, which would have turned a loss of 0.017977 into 0.17977 without any indication that anything was wrong. Go ahead, ask me how I know.

Luckily I'm a resourceful guy. Back in 2016 when I was still working heavily with 8-bit PIC microcontrollers I wrote an ftoa that walks the fractional part digit by digit, and you can see the original in EUSART.c from my PIC16 projects. The modded version is in the repo linked below with the full Baochip GPT. The two lines that add a half step before the split are load bearing, because without them each value *= 10.0f drifts slightly low in float32 and the last digit comes out one under, so 0.017977 prints as 0.017976 and 0.046150 prints as 0.046149. I only caught it because I was comparing against a desktop build digit by digit.

So then the next question was memory.

Where the memory goes

The model itself is 108 KB. Training costs nine times that, because Adam carries two moment estimates alongside the gradients and the backward pass needs every intermediate activation held in memory. The single largest item in the cache is the post softmax attention matrix at 128 KB, which is mostly zeros since everything above the diagonal is masked out. So that means half the SRAM is untouched.

Then someone reading this will be like "you said 37KB". Set that against the table for a moment. Everything that implements this, attention and the causal mask and the full backward pass and Adam and the gradient checker, along with the SDK drivers and libm and the entire software floating point runtime that makes any of it possible on this core, comes to 37KB. The parameters alone are 108 KB, so the instructions that perform backpropagation take a third of the space of the numbers they operate on, and about a twelfth of the activation cache they write into. Training is expensive in memory and remarkably cheap in code, and I myself wasn't fully aware of how lopsided that ratio is until I started writing stuff by hand.

So then we have to start thinking about how long the model takes to train.

Where the time goes

A training step takes 3487 ms. At 350 MHz that is roughly 1.22 billion cycles for one gradient update on 27,680 parameters. I instrumented the transcendental calls to find out where it was going:

Which turns out not to be the answer. Even at a generous fifteen hundred cycles apiece those calls are about eight percent of the run. Building a tanh lookup table, which was my first instinct, would have bought almost nothing. The other ninety two percent is plain multiplication and addition. One training step is about 4.6 million float operations, roughly 1.37 million forward, 3.05 million backward, 220 thousand in Adam. That works out to around 265 cycles per float operation once you include call overhead and memory traffic.The most important measurement of the whole project came from changing the optimisation level. On my desktop, -Os to -O2 gave a 1.63x speedup but on the Baochip it gave one percent, so 3522 ms became 3487 ms.

That gap is the entire story of soft float. On a machine with an FPU the compiler has instructions to schedule, reorder and vectorise. When every operation is an opaque call into a precompiled library the optimiser can only tidy the loop arithmetic around the calls, and the calls are the cost. There is no compiler flag that makes software floating point fast.

Verifying that it actually worked

This is the part I care about most, and it is the part most machine learning demonstrations skip or gloss over. A loss curve going down is not proof of correctness. A subtly wrong backward pass will still produce a falling loss, just a worse one, and you will never know that. So before the model trains a single step, the chip verifies its own gradients. It perturbs each parameter block along a random direction, measures the actual change in loss, and compares that against the analytic gradient projected onto the same direction. Averaging over a direction rather than checking single elements removes the float32 noise that makes element wise checks useless at this precision.

If any block disagrees the program halts and refuses to train, because a training run built on a wrong derivative is worse than no training run at all, then there is the result I did not expect. Because I replaced rand() with xorshift32 to solve a linker error, the desktop build and the Baochip build started from bit identical random weights, so I could compare them directly:

Os we get forty six seconds per epoch, thirteen batches each, steady throughout. 4610 seconds total. Two baselines print at startup so the numbers mean something, guessing uniformly across 28 symbols scores 3.332205, and predicting from character frequency alone scores 2.557692. A model that finishes above the second one has learned nothing beyond which letters are common/The interesting window is epoch 30 to 40, where the loss falls from 0.316 to 0.031. That's Ten fold in ten epochs, about eight minutes, and it is exactly where the output stops being noise:

epoch 10    the ig small cathe the dog
epoch 20    the mat. sat on the mat. s
epoch 30    the mat. at. athe t. t sma
epoch 40    the mat. a cat ran. the bi
epoch 100   the mat. a cat ran. the bi

Nothing changes after epoch 40. Everything past that is the loss getting smaller without the behaviour changing, which is the signature of a model that has finished learning and started memorising, and that is a real limitation of running 27,680 parameters against a 248 character corpus. Forty epochs would have taken thirty one minutes and produced the same text. The finished weights go into ReRAM, 108 KB written once after training and never inside the loop, because ReRAM has finite write endurance and Adam would destroy it. Pull the power, plug it back in, and the chip generates text immediately without retraining.

IEEE 754 single precision is fully specified and libgcc's software implementation is correct, so the two machines perform nearly the identical computation despite one of them having no floating point hardware. Then they separate, and that is more interesting than the match because of one unit in the last place difference, The same kind visible in norm2_g above, gets picked up by Adam and amplified until the two runs are walking different paths down the same loss surface. They still land in the same place though, I guess that's just gradient descent being chaotic.

The ReRAM in LLMs Bits

There is something almost funny about where those weights ended up.

I spent an hour running 4.6 million software emulated float operations per step on a core with no floating point hardware, then wrote the finished parameters into ReRAM, which is the same base technology a substantial body of research is trying to turn into a matrix multiplier.

A ReRAM crossbar stores a weight as a conductance rather than a number, so if you drive the rows with voltages representing your input vector, the current summing on each column is the dot product, computed by Ohm's law and Kirchhoff's current law instead of an instruction sequence. PRIME, from ISCA, proposed configuring parts of a ReRAM main memory as neural network accelerators on exactly this basis. There is a whole line of work since, on bit sparsity in the crossbar, on reconfigurable in situ accelerators, on conductance aware quantization to deal with real cells only supporting a limited number of discrete conductance states.

The Baochip-1x does not do any of that. Its ReRAM is byte addressable non volatile memory on the normal memory bus. No sense amplifiers arranged for analog readout or any peripheral circuitry that turns a memory array into a multiplier. bunnie put ReRAM on this chip because it is a good replacement for flash on 22nm, not because he was building an accelerator, and nothing I did exploits the crossbar. But the weights of my transformer are sitting in resistive memory cells right now, as stored numbers, having been computed one soft float call at a time by a core a few millimetres away. The technology holding them is the technology people are trying to make compute them. Strange thing to notice while watching a serial console print a loss value every forty six seconds.

What is a Language Model Anyway? 

So what I've built here is by all defintions a language model. It is a real transformer performing real gradient descent, verified against numerical differentiation on the target hardware, on a processor that cannot multiply two floating point numbers without calling a function to do it. But it is also very small, trained on very little text, and trained past the point where learning is more memorisation than anything else. I'm telling you that rather than making you find it.

Which is the part that interests me, actually. It is a language model.

It models a distribution over tokens and predicts the next one, and that is the definition. There is no clause specifying how many parameters you need before the words apply to you. A go-kart is a vehicle. Nobody argues about that and nobody confuses it with an F1 car either, because the word is about what the thing is and not about how fast it goes. But I know that when I say "language model", the definition is going to get very precise very quickly. Then that got me thinking. These frontier AI companies can ship a model, call what it does reasoning, and nobody stops the sentence to ask where the line is. And then there is AGI, which nobody can define at all. There isn't a threshold or test everyone accepts at least that I or anyone outside the frontier labs knows of, and no measurement you could run tomorrow that would settle it. It is the least defined term in the industry and the one that moves the most capital right now, and I do not think those two facts are unrelated.

Look at the goalposts. Chess was going to be "the moment", until it happened, and then chess was just search. Go was going to be "the moment" because it needed intuition, until it happened, and then it was pattern matching over a large space. The Turing test stood for fifty years and got retired the moment machines started passing it, many people struggle to tell LLM text from human texts, if Alan Turing were alive today and talked to a frontier model, what would he say? Every time a system cleared a bar, the bar was reclassified as something that never really counted.

And the direction it moves is the tell. Surprising output becomes evidence of progress toward general intelligence. A failure a child would not make means nobody ever claimed that, it is only a language model after all. So the word stretches when it needs to cover more and shrinks when it needs to cover less, and none of that stretching is being done by evidence. I'm not complaining about it, I just think it is worth saying out loud as I dabble in these systems. 

Anyway. the whole thing is on GitHub, and if you find something I got wrong I would rather hear it than not:

ArmstrongSubero/AI-from-cratch-in-C: Neural networks from scratch in C, from one perceptron to a working GPT. Code for the book AI from Scratch in C.

Armstrong Subero is an embedded systems engineer and published author with Apress/Springer. He builds the Rovari RISC-V education platform from Trinidad and Tobago.