The words the record uses, defined for a reader who does not build any of this. Entries link here on first use.
A fixed set of test questions used to compare models. Useful, gamed often, and frequently misread.
Benchmarks make claims comparable: the same exam, the same marking, several systems. They have two standing weaknesses. Test questions leak into training data, so a model may have seen the answers; and a score on a fixed set of questions says little about behavior on the messy problems people actually bring. Read a benchmark result as evidence about that benchmark, and no more than that.
A model trained on very large amounts of text to predict what word comes next, and used to write and answer.
The task these systems are trained on is narrow: given some text, guess what follows. Done at sufficient scale, that one task produces something that can summarize a document, draft an email, translate a paragraph, or write code. The output is fluent by construction, which is the trap. A large language model has no separate store of facts to check itself against, so a wrong answer arrives in the same confident prose as a right one.
A program whose behavior was learned from examples rather than written out as rules by a person.
Ordinary software follows instructions somebody typed. A model is built the other way around: it is shown a great many examples and adjusts itself until it produces answers that fit them. What comes out is a large file of numbers, not a list of steps a person could read. That is why the people who build models can describe how one was made in detail and still be unable to say why it gave a particular answer.
A model whose trained parameters are published, so anyone can download and run it on their own hardware.
Open weights are not the same as open source. The numbers are published, and often a license setting out what may be done with them, but the training data and the code that produced them usually are not, so the result cannot be independently reproduced. What the practice does change is control: a downloaded model runs without an internet connection, cannot be withdrawn by the company that made it, and cannot be modified after the fact.
One of the adjustable numbers inside a model. Modern language models have billions of them.
A parameter is a dial. During training the system turns billions of them a little at a time, keeping the settings that make its answers better and discarding the ones that make them worse. Counts are quoted as a rough measure of size: GPT-3 was described as having 175 billion parameters. Size is not the same as usefulness. A smaller model trained on better data often does the job of a larger one at a fraction of the running cost.
Training by trial, error and reward rather than by copying labelled examples.
Instead of being shown the right answer, the system tries something, receives a score, and adjusts. It is how AlphaGo improved past the human games it started from, by playing millions of games against copies of itself. A variant is used on chatbots: people rank competing answers, those rankings train a scoring model, and the scoring model is used to tune the chatbot toward replies people preferred.
The process of showing a model examples and adjusting its parameters until its answers improve.
Training is the expensive part. The system makes a prediction, is told how far off it was, and nudges its parameters toward a better answer. Repeat that several trillion times, across thousands of processors running for weeks, and the result is a model. Nothing is memorized in the way a database stores a record, which is why a model can answer a question nobody asked it during training, and also why it can invent an answer with complete confidence.
The network design, published by Google in 2017, that almost every large language model is built on.
Earlier systems read a sentence one word at a time, carrying a summary forward, and tended to lose the thread over long passages. The transformer lets every word weigh every other word directly, however far apart they sit, and does all of those comparisons at once rather than in order. That made the work easy to spread across many processors, which is what allowed models to grow to their present size. See the entry for the paper that introduced it.