Using a language-model API can make the system feel almost magical. You send text and receive useful text. NanoLLM started because I wanted to remove that feeling and understand the machinery between those two events.
The project is intentionally small. The goal is not to compete with production models. The goal is to make every major step visible enough to inspect, change, and break.
Text becomes numbers
The first important lesson is that a model never sees words. A tokenizer turns text into a sequence of identifiers. That decision affects vocabulary size, context length, training efficiency, and how the model handles unfamiliar text. A poor tokenizer creates problems that no amount of model depth can fully hide.
Each token then becomes an embedding: a learned vector that gives the network something continuous to operate on. Position information matters too, because a sequence without order loses its meaning.
Attention is routing
Attention became clearer when I stopped describing it as “the model focusing.” Mechanically, it is a learned routing system. Queries, keys, and values create a weighted exchange of information between positions in the sequence. Multiple heads allow different relationships to be represented at the same time.
Implementing the tensor shapes by hand was more educational than reading another diagram. Most bugs were not theoretical. They were mismatched dimensions, incorrect masking, unstable loss, or data moving through the model in an unexpected form.
Training changes the perspective
Training a small model makes limitations obvious. Data quality matters. Batch size changes the noise of optimization. Learning rate can make a model improve, stall, or collapse. A decreasing loss does not automatically mean useful generation.
NanoLLM taught me to treat modern AI as an engineered system, not a mysterious intelligence. There are layers of choices—data, representation, architecture, optimization, and inference—and each layer creates tradeoffs. Once those layers are visible, it becomes much easier to reason about what a larger model can and cannot do.