{"id":488664,"date":"2026-07-27T07:38:49","date_gmt":"2026-07-27T07:38:49","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=488664"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=488664","title":{"rendered":"ML Without Magic: Building a Tiny Language Model in Pure Node.js and Watching Every Weight Change"},"content":{"rendered":"
\n
\n

Tokenization \u2192 embeddings \u2192 causal Transformer \u2192 LM head \u2192 softmax \u2192 loss \u2192 backpropagation. No TensorFlow, no PyTorch, and no hidden autograd.<\/p>\n<\/blockquote>\n

Most explanations of language models present correct formulas but hide the path between them inside a framework. I wanted the opposite: one small scenario where every scalar is visible and where the terminal clearly shows incorrect answers before learning and correct answers after it.<\/p>\n

The project now has one command:<\/p>\n

node src\/train.js --generalize --adaptive-teach<\/code>
<\/a><\/div><\/pre>\n
It requires Node.js 18.19+ and has no dependencies.<\/p>\n
A real excerpt from logs\/training-log.txt<\/code>, showing the AFTER and DELTA matrices for one FFN layer:<\/p>\n
\"AFTER<\/p>\n
AFTER and DELTA weight matrices from logs\/training-log.txt<\/figcaption><\/div>\n<\/figure>\n
The result first<\/h3>\n
The model is queried immediately after random initialization:<\/p>\n
BEFORE TRAINING \u2014 random, usually wrong answers> can human read ?  model:    ? <unk> ...  expected: human can read.  [WRONG]> can fish swim ?  model:    ? <unk> ...  expected: fish can swim.   [WRONG]> can cat read ?  model:    ? <unk> ...  expected: cat cannot read. [WRONG]<\/code>
<\/a><\/div><\/pre>\n
After pre-training, SFT, and adaptive SFT, the same model produces:<\/p>\n
FINAL ANSWERS AFTER ADAPTIVE SFT> can human read ?  model:    human can read.  [CORRECT]> can fish swim ?  model:    fish can swim.   [CORRECT]> can bird fly ?  model:    bird can fly.    [CORRECT]> can cat read ?  model:    cat cannot read. [CORRECT]Rehearsal controls preserved: 14\/14.Stable criterion reached 11 times in a row.<\/code>
<\/a><\/div><\/pre>\n
The initial text varies because initialization is random. The final acceptance criterion does not: all answers must be correct, every target token must have at least 95% probability, and the complete check must pass more than ten times consecutively.<\/p>\n
What remains after removing the extra modes<\/h3>\n
The code previously contained several debug and training modes. They were useful while experimenting but obscured the main idea. The final version keeps one educational pipeline:<\/p>\n
text \u2192 word tokenization \u2192 token IDs     \u2192 token + position embeddings     \u2192 two causal Transformer blocks        \u2192 multi-head self-attention        \u2192 two-hidden-layer FFN     \u2192 LM head \u2192 softmax \u2192 next-token probabilities     \u2192 cross-entropy \u2192 backpropagation \u2192 Adam<\/code>
<\/a><\/div><\/pre>\n
train.js<\/code> now reads as one story rather than a command-line framework.<\/p>\n
A scalar builds the computation graph<\/h3>\n
Every number participating in learning is a Value<\/code>:<\/p>\n
class Value {  constructor(data, children = [], backward = () => {}) {    this.data = data;    this.grad = 0;    this.children = children;    this._backward = backward;  }}<\/code>
<\/a><\/div><\/pre>\n
For multiplication:<\/p>\n
y = a \u00d7 bdy\/da = bdy\/db = a<\/code>
<\/a><\/div><\/pre>\n
The operation stores these local derivatives. backward()<\/code> sorts the graph topologically and applies the chain rule from the final loss back to embeddings and weights.<\/p>\n
A neuron is literally an object<\/h3>\n
The neuron formula is not hidden behind a tensor API:<\/p>\n
output = activation(sum(input[i] \u00d7 weight[i]) + bias)<\/code>
<\/a><\/div><\/pre>\n
Its implementation follows the formula:<\/p>\n
forward(input) {  let output = sum(    input.map((value, i) => value.mul(this.weights[i]))  );  if (this.useBias) output = output.add(this.bias);  if (this.activation === 'relu') return output.relu();  return output;}<\/code>
<\/a><\/div><\/pre>\n
A Linear<\/code> layer is just an array of neurons receiving the same input. This is slower than matrix multiplication but far easier to inspect.<\/p>\n
Embeddings and order<\/h3>\n
Each token ID selects one trainable vector:<\/p>\n
token representation = tokenEmbedding[id] + positionEmbedding[position]<\/code>
<\/a><\/div><\/pre>\n
Embeddings contain random values initially. They acquire useful relations only because gradients repeatedly change them in training contexts. No meaning<\/code> property is assigned to cat<\/code>, read<\/code>, or cannot<\/code>.<\/p>\n
Self-attention without shorthand<\/h3>\n
For every token:<\/p>\n
Q = X \u00d7 WqK = X \u00d7 WkV = X \u00d7 Wvscore = dot(Q, K) \/ sqrt(headSize)attention = softmax(score)output = attention \u00d7 V<\/code>
<\/a><\/div><\/pre>\n
The implementation loops only while past <= position<\/code>. That is the causal mask: the model can attend to the current token and its history but never to a future target.<\/p>\n
After attention, every token passes through a two-hidden-layer feed-forward network:<\/p>\n
dModel \u2192 hidden ReLU \u2192 hidden ReLU \u2192 dModel<\/code>
<\/a><\/div><\/pre>\n
LayerNorm and residual paths preserve stable information flow around attention and FFN.<\/p>\n
The complete learning step<\/h3>\n
The most important code in the project is only a few lines:<\/p>\n
function learnOneToken({ model, optimizer, input, targetId }) {  const loss = model.loss(input, targetId);  optimizer.zeroGrad();  loss.backward();  optimizer.step();  return loss.data;}<\/code>
<\/a><\/div><\/pre>\n
The loss is ordinary next-token cross-entropy:<\/p>\n
loss = -log(P(target | previous tokens))<\/code>
<\/a><\/div><\/pre>\n
If the correct token has low probability, loss is large. Backpropagation computes dLoss\/dWeight<\/code>; Adam changes each parameter; the next forward pass gives a different distribution.<\/p>\n
Phase 1: pre-training<\/h3>\n
The tiny world contains 14 ability relations:<\/p>\n
human can read .fish can swim .bird can fly .dog cannot read .<\/code>
<\/a><\/div><\/pre>\n
The cat + read<\/code> relation is missing deliberately. Pre-training samples positions from this text and learns ordinary next-token prediction.<\/p>\n
Phase 2: SFT<\/h3>\n
The same relations are converted into 42 prompt-answer examples:<\/p>\n
can fish swim ?is fish able to swim ?does fish know how to swim ?<\/code>
<\/a><\/div><\/pre>\n
Only answer tokens contribute to SFT loss. The implementation visits every pair and every answer position on each epoch, making the training loop deterministic and readable.<\/p>\n
Phase 3: adaptive SFT<\/h3>\n
The missing answer is represented only by target tokens:<\/p>\n
['cat', 'cannot', 'read', '.']<\/code>
<\/a><\/div><\/pre>\n
Six question variants receive those targets. This is direct supervision: the model did not discover a zoological fact on its own. The teacher introduced the fact through loss, and backpropagation distributed that information across embeddings, attention, FFN, LayerNorm, and the LM head.<\/p>\n
Why not stop after one correct answer? Because one generation can be fragile. The loop continues until every target token exceeds 95% probability and the whole evaluation succeeds 11 times in a row.<\/p>\n
Catastrophic forgetting and rehearsal<\/h3>\n
An early implementation trained only the six new cat prompts. It successfully learned the new answer and destroyed old behavior:<\/p>\n
can human read ? \u2192 cat cannot read.can fish swim ?  \u2192 cat cannot read.<\/code>
<\/a><\/div><\/pre>\n
That is catastrophic forgetting in miniature. The fix is rehearsal: adaptive epochs also repeat the 14 older can ... ?<\/code> examples. The final criterion evaluates both new and old examples, so training cannot finish by overwriting everything with one response.<\/p>\n
The log is always written<\/h3>\n
The command automatically creates:<\/p>\n
logs\/training-log.txt<\/code>
<\/a><\/div><\/pre>\n
It is a sequential ASCII diagram rather than a raw JSON dump. It includes every forward\/loss\/backward\/update event, followed by the complete matrices at three checkpoints:<\/p>\n
initial random matrices        |        vmatrices after pre-training + SFT        |        vfinal matrices after adaptive SFT<\/code>
<\/a><\/div><\/pre>\n
For every transition, the log prints the AFTER matrix and its exact DELTA matrix. Linear rows are named neuron[n]<\/code>, columns are named weight[n]<\/code>, and biases are shown beside their neuron. It also points out the largest concrete change as layer \/ neuron \/ weight: before -> after -> delta<\/code>.<\/p>\n
How close is it to a production LLM?<\/h3>\n
The architecture and learning rule are real; the scale is intentionally tiny.<\/p>\n
\n
\n\n\n\n\n\n\n\n\n\n
\n
This model<\/p>\n<\/th>\n
\n
Production model<\/p>\n<\/th>\n<\/tr>\n
\n
24 word tokens<\/p>\n<\/td>\n
\n
Large subword\/byte vocabulary<\/p>\n<\/td>\n<\/tr>\n
\n
2,160 parameters<\/p>\n<\/td>\n
\n
Millions or billions<\/p>\n<\/td>\n<\/tr>\n
\n
Two Transformer blocks<\/p>\n<\/td>\n
\n
Tens or hundreds<\/p>\n<\/td>\n<\/tr>\n
\n
Scalar JavaScript graph<\/p>\n<\/td>\n
\n
Batched tensor graph on accelerators<\/p>\n<\/td>\n<\/tr>\n
\n
Small structured corpus<\/p>\n<\/td>\n
\n
Massive curated datasets<\/p>\n<\/td>\n<\/tr>\n
\n
Narrow trained behavior<\/p>\n<\/td>\n
\n
Broad language and reasoning<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
The project is not a GPT competitor. It is a causal language model reduced until the complete path fits in one repository and one mental model:<\/p>\n
token \u2192 embedding \u2192 attention \u2192 FFN \u2192 probability      \u2192 loss \u2192 gradient \u2192 updated weight \u2192 changed answer<\/code>
<\/a><\/div><\/pre>\n
That path is the point. Once it is visible, frameworks stop looking magical: they execute the same classes of operations at a scale and speed this scalar implementation deliberately avoids.<\/p>\n
Repository: tiny-language-model-neuro-js<\/strong><\/a>.<\/p>\n<\/div>\n
\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1063326\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Tokenization \u2192 embeddings \u2192 causal Transformer \u2192 LM head \u2192 softmax \u2192 loss \u2192 backpropagation. No TensorFlow, no PyTorch, and no hidden autograd.Most explanations of language models present correct formulas but hide the path between them inside a framework. I wanted the opposite: one small scenario where every scalar is visible and where the terminal clearly shows incorrect answers before learning and correct answers after it.The project now has one command:node src\/train.js —generalize —adaptive-teachIt requires Node.js 18.19+ and has no dependencies.A real excerpt from logs\/training-log.txt, showing the AFTER and DELTA matrices for one FFN layer:AFTER and DELTA weight matrices from logs\/training-log.txtThe result firstThe model is queried immediately after random initialization:BEFORE TRAINING \u2014 random, usually wrong answers> can human read ?  model:    ? <unk> …  expected: human can read.  [WRONG]> can fish swim ?  model:    ? <unk> …  expected: fish can swim.   [WRONG]> can cat read ?  model:    ? <unk> …  expected: cat cannot read. [WRONG]After pre-training, SFT, and adaptive SFT, the same model produces:FINAL ANSWERS AFTER ADAPTIVE SFT> can human read ?  model:    human can read.  [CORRECT]> can fish swim ?  model:    fish can swim.   [CORRECT]> can bird fly ?  model:    bird can fly.    [CORRECT]> can cat read ?  model:    cat cannot read. [CORRECT]Rehearsal controls preserved: 14\/14.Stable criterion reached 11 times in a row.The initial text varies because initialization is random. The final acceptance criterion does not: all answers must be correct, every target token must have at least 95% probability, and the complete check must pass more than ten times consecutively.What remains after removing the extra modesThe code previously contained several debug and training modes. They were useful while experimenting but obscured the main idea. The final version keeps one educational pipeline:text \u2192 word tokenization \u2192 token IDs     \u2192 token + position embeddings     \u2192 two causal Transformer blocks        \u2192 multi-head self-attention        \u2192 two-hidden-layer FFN     \u2192 LM head \u2192 softmax \u2192 next-token probabilities     \u2192 cross-entropy \u2192 backpropagation \u2192 Adamtrain.js now reads as one story rather than a command-line framework.A scalar builds the computation graphEvery number participating in learning is a Value:class Value {  constructor(data, children = [], backward = () => {}) {    this.data = data;    this.grad = 0;    this.children = children;    this._backward = backward;  }}For multiplication:y = a \u00d7 bdy\/da = bdy\/db = aThe operation stores these local derivatives. backward() sorts the graph topologically and applies the chain rule from the final loss back to embeddings and weights.A neuron is literally an objectThe neuron formula is not hidden behind a tensor API:output = activation(sum(input[i] \u00d7 weight[i]) + bias)Its implementation follows the formula:forward(input) {  let output = sum(    input.map((value, i) => value.mul(this.weights[i]))  );  if (this.useBias) output = output.add(this.bias);  if (this.activation === ‘relu’) return output.relu();  return output;}A Linear layer is just an array of neurons receiving the same input. This is slower than matrix multiplication but far easier to inspect.Embeddings and orderEach token ID selects one trainable vector:token representation = tokenEmbedding[id] + positionEmbedding[position]Embeddings contain random values initially. They acquire useful relations only because gradients repeatedly change them in training contexts. No meaning property is assigned to cat, read, or cannot.Self-attention without shorthandFor every token:Q = X \u00d7 WqK = X \u00d7 WkV = X \u00d7 Wvscore = dot(Q, K) \/ sqrt(headSize)attention = softmax(score)output = attention \u00d7 VThe implementation loops only while past <= position. That is the causal mask: the model can attend to the current token and its history but never to a future target.After attention, every token passes through a two-hidden-layer feed-forward network:dModel \u2192 hidden ReLU \u2192 hidden ReLU \u2192 dModelLayerNorm and residual paths preserve stable information flow around attention and FFN.The complete learning stepThe most important code in the project is only a few lines:function learnOneToken({ model, optimizer, input, targetId }) {  const loss = model.loss(input, targetId);  optimizer.zeroGrad();  loss.backward();  optimizer.step();  return loss.data;}The loss is ordinary next-token cross-entropy:loss = -log(P(target | previous tokens))If the correct token has low probability, loss is large. Backpropagation computes dLoss\/dWeight; Adam changes each parameter; the next forward pass gives a different distribution.Phase 1: pre-trainingThe tiny world contains 14 ability relations:human can read .fish can swim .bird can fly .dog cannot read .The cat + read relation is missing deliberately. Pre-training samples positions from this text and learns ordinary next-token prediction.Phase 2: SFTThe same relations are converted into 42 prompt-answer examples:can fish swim ?is fish able to swim ?does fish know how to swim ?Only answer tokens contribute to SFT loss. The implementation visits every pair and every answer position on each epoch, making the training loop deterministic and readable.Phase 3: adaptive SFTThe missing answer is represented only by target tokens:[‘cat’, ‘cannot’, ‘read’, ‘.’]Six question variants receive those targets. This is direct supervision: the model did not discover a zoological fact on its own. The teacher introduced the fact through loss, and backpropagation distributed that information across embeddings, attention, FFN, LayerNorm, and the LM head.Why not stop after one correct answer? Because one generation can be fragile. The loop continues until every target token exceeds 95% probability and the whole evaluation succeeds 11 times in a row.Catastrophic forgetting and rehearsalAn early implementation trained only the six new cat prompts. It successfully learned the new answer and destroyed old behavior:can human read ? \u2192 cat cannot read.can fish swim ?  \u2192 cat cannot read.That is catastrophic forgetting in miniature. The fix is rehearsal: adaptive epochs also repeat the 14 older can … ? examples. The final criterion evaluates both new and old examples, so training cannot finish by overwriting everything with one response.The log is always writtenThe command automatically creates:logs\/training-log.txtIt is a sequential ASCII diagram rather than a raw JSON dump. It includes every forward\/loss\/backward\/update event, followed by the complete matrices at three checkpoints:initial random matrices        |        vmatrices after pre-training + SFT        |        vfinal matrices after adaptive SFTFor every transition, the log prints the AFTER matrix and its exact DELTA matrix. Linear rows are named neuron[n], columns are named weight[n], and biases are shown beside their neuron. It also points out the largest concrete change as layer \/ neuron \/ weight: before -> after -> delta.How close is it to a production LLM?The architecture and learning rule are real; the scale is intentionally tiny.This modelProduction model24 word tokensLarge subword\/byte vocabulary2,160 parametersMillions or billionsTwo Transformer blocksTens or hundredsScalar JavaScript graphBatched tensor graph on acceleratorsSmall structured corpusMassive curated datasetsNarrow trained behaviorBroad language and reasoningThe project is not a GPT competitor. It is a causal language model reduced until the complete path fits in one repository and one mental model:token \u2192 embedding \u2192 attention \u2192 FFN \u2192 probability      \u2192 loss \u2192 gradient \u2192 updated weight \u2192 changed answerThat path is the point. Once it is visible, frameworks stop looking magical: they execute the same classes of operations at a scale and speed this scalar implementation deliberately avoids.Repository: tiny-language-model-neuro-js.\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1063326\/<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-488664","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/488664","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=488664"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/488664\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=488664"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=488664"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=488664"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}