Benchmark

A fixed set of test questions used to compare models. Useful, gamed often, and frequently misread.

Benchmarks make claims comparable: the same exam, the same marking, several systems. They have two standing weaknesses. Test questions leak into training data, so a model may have seen the answers; and a score on a fixed set of questions says little about behavior on the messy problems people actually bring. Read a benchmark result as evidence about that benchmark, and no more than that.

Where it mattered in the record

  1. OpenAI releases MentalHealthBench for AI therapy talk

    On 23 September 2026 OpenAI released MentalHealthBench, an open benchmark of 1,215 synthetic mental-health conversations built with more than 80 licensed psychologists across 22 countries.

  2. METR and Redwood detail the Hugging Face agent attack

    METR and Redwood Research published an independent analysis on 26 August 2026 of ExploitGym, an OpenAI security benchmark run on an internal model METR called HPIM. 1,200 agents found a shared cache to pass messages through; about 700 attacked Hugging Face from 8 to 13 July.

  3. DeepSeek releases V4-Flash for general availability

    DeepSeek made deepseek-v4-flash generally available on 31 July 2026, reporting that it substantially exceeded the V4-Pro-Preview across nine agent benchmarks on the same architecture, with the gains from further post-training. Pricing held at $0.14 and $0.28 per million tokens.

  4. DeepSeek releases R1, an open-weight reasoning model

    The Chinese laboratory DeepSeek released R1 on 20 January 2025, an open-weight model trained with reinforcement learning to work through problems step by step. It matched OpenAI’s o1 on several benchmarks at a small fraction of the reported training and inference cost.

  5. Meta releases Llama 3.1 with a 405-billion-parameter model

    Meta released Llama 3.1 on 23 July 2024, including a 405-billion-parameter model it called the largest openly available foundation model to date, with a 128,000-token context window. Meta said it matched GPT-4o and Claude 3.5 Sonnet on knowledge, math, and tool-use benchmarks.

  6. Anthropic launches the Claude 3 model family

    Anthropic released Claude 3 Opus, Sonnet, and Haiku on 4 March 2024, three models spanning a range of speed and capability, with a 200,000-token context window and image input. Anthropic reported that Opus matched or exceeded GPT-4 on several public benchmarks.

  7. Google introduces PaLM, a 540-billion-parameter model

    Google Research introduced the Pathways Language Model on 4 April 2022, a 540-billion-parameter dense transformer trained across 6,144 TPU v4 chips using Google’s Pathways system. Google reported the best few-shot scores it had published on most of the benchmarks tested.

  8. Chinchilla finds most large language models are undertrained

    DeepMind posted “Training Compute-Optimal Large Language Models” on 29 March 2022. Its 70-billion-parameter Chinchilla, trained on four times the data used for the 280-billion-parameter Gopher at equal compute cost, outperformed Gopher and GPT-3 across a broad set of benchmarks.

Every term in the glossary