Transformer

The network design, published by Google in 2017, that almost every large language model is built on.

Earlier systems read a sentence one word at a time, carrying a summary forward, and tended to lose the thread over long passages. The transformer lets every word weigh every other word directly, however far apart they sit, and does all of those comparisons at once rather than in order. That made the work easy to spread across many processors, which is what allowed models to grow to their present size. See the entry for the paper that introduced it.

Where it mattered in the record

  1. Aleph Alpha releases open-weight model Kolibri

    On 3 October 2026 Aleph Alpha released Kolibri, an English-German mixture-of-experts Transformer with 78.1B total parameters and 3.46B active, under Apache 2.0 on Hugging Face. Context reaches 1M tokens, and Merlin-Arthur withholds answers the context does not support.

  2. Google introduces PaLM, a 540-billion-parameter model

    Google Research introduced the Pathways Language Model on 4 April 2022, a 540-billion-parameter dense transformer trained across 6,144 TPU v4 chips using Google’s Pathways system. Google reported the best few-shot scores it had published on most of the benchmarks tested.

  3. Microsoft and Nvidia unveil a 530-billion-parameter model

    Microsoft and Nvidia announced Megatron-Turing NLG on 11 October 2021, a 530-billion-parameter transformer language model. It was trained with DeepSpeed and Megatron-LM on 560 Nvidia DGX A100 servers, drawing largely on the Pile and Common Crawl web data.

  4. Google publishes BERT, a bidirectional language model

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova published BERT on 11 October 2018, a transformer that pretrains representations of text by reading in both directions, then fine-tuned for particular tasks. Google released the code and weights the following month.

  5. OpenAI publishes the GPT-1 paper on pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever published “Improving Language Understanding by Generative Pre-Training” on 11 June 2018, describing a transformer pretrained on unlabeled text and then fine-tuned for particular tasks.

  6. Google researchers publish “Attention Is All You Need”

    Eight researchers at Google posted a paper to arXiv describing the transformer, a network built on attention alone, with no recurrence. It reported 28.4 BLEU on the WMT 2014 English-to-German task, trained on eight GPUs in three and a half days.

Every term in the glossary