Research
What we're making next, and what we've learned so far training small language models from scratch. Every number here comes from our own lab notebook.
Coming next
Our next release is a whole family at once.
The Loom 4.5 family
Coming soonSeven models, from 11 million to 130 million parameters, all from one main training run. They are built to read the text you give them and pull out the answer, use the tools you list, follow small rules from a few examples, and say how sure they are.
- One run, five models. While Crucible 4.5 trains, it also learns to answer from partway down. Its first 8, 12, 16 and 24 layers each work as a complete smaller model.
- Proved before it's spent. A one-hour run on a small model tests the training data first. The eight-hour run only starts once that passes.
- Bars set in advance. The tests and the scores each model must reach are written down before training starts, and never lowered.
What we've learned
Short notes from our experiments, newest first. Some of these are wins; most are things that surprised us.
How we work
The rules we hold ourselves to, learned the hard way.
From scratch
Every Loom starts from random weights. We never fine-tune someone else's model.
Tests first
We write the tests and the pass marks before training, and keep them out of the training data.
Report the misses
When a model misses its bar, we say so on its card instead of moving the bar.
Write it down
Every run, number and mistake goes into the lab notebook, so nothing is learned twice.