Large language model

A model trained on very large amounts of text to predict what word comes next, and used to write and answer.

The task these systems are trained on is narrow: given some text, guess what follows. Done at sufficient scale, that one task produces something that can summarize a document, draft an email, translate a paragraph, or write code. The output is fluent by construction, which is the trap. A large language model has no separate store of facts to check itself against, so a wrong answer arrives in the same confident prose as a right one.

Where it mattered in the record

  1. Meta releases Llama 3.1 with a 405-billion-parameter model

    Meta released Llama 3.1 on 23 July 2024, including a 405-billion-parameter model it called the largest openly available foundation model to date, with a 128,000-token context window. Meta said it matched GPT-4o and Claude 3.5 Sonnet on knowledge, math, and tool-use benchmarks.

  2. Meta releases Llama 3 under an open license

    Meta released the first Llama 3 models, at 8 billion and 70 billion parameters, on 18 April 2024 under a license permitting most commercial use. Meta said the 70B model beat comparable models, including Claude 3 Sonnet, on human evaluations across twelve use cases.

  3. Anthropic launches the Claude 3 model family

    Anthropic released Claude 3 Opus, Sonnet, and Haiku on 4 March 2024, three models spanning a range of speed and capability, with a 200,000-token context window and image input. Anthropic reported that Opus matched or exceeded GPT-4 on several public benchmarks.

  4. OpenAI releases ChatGPT as a free research preview

    OpenAI published ChatGPT on 30 November 2022: a conversational interface to a model trained with human feedback and described as a sibling of InstructGPT. It was free during the research preview and reached through a web page, with no API key and no waitlist.

  5. Meta withdraws its Galactica science model after three days

    Meta AI released Galactica on 15 November 2022, a language model trained on scientific text and intended to help write papers and summaries, with a public demo. Users showed it producing fluent but false claims and invented citations, and Meta took the demo down on 17 November.

  6. BigScience releases BLOOM with 176 billion parameters

    The BigScience collective, hundreds of volunteer researchers coordinated around Hugging Face and using France’s Jean Zay supercomputer, released BLOOM on 12 July 2022. The 176-billion-parameter model wrote 46 natural languages and 13 programming languages.

  7. Google introduces PaLM, a 540-billion-parameter model

    Google Research introduced the Pathways Language Model on 4 April 2022, a 540-billion-parameter dense transformer trained across 6,144 TPU v4 chips using Google’s Pathways system. Google reported the best few-shot scores it had published on most of the benchmarks tested.

  8. Chinchilla finds most large language models are undertrained

    DeepMind posted “Training Compute-Optimal Large Language Models” on 29 March 2022. Its 70-billion-parameter Chinchilla, trained on four times the data used for the 280-billion-parameter Gopher at equal compute cost, outperformed Gopher and GPT-3 across a broad set of benchmarks.

Every term in the glossary