Mixture of experts

A design where a model holds many specialist sub-models and consults only a few for each token.

In an ordinary large model every parameter is used for every word. In a mixture-of-experts design, the model holds dozens of “experts” and routes each token to the two or three most relevant, so total size can be enormous while the cost per answer stays modest. The price is complexity: training must also teach the router which expert handles what.

Where it mattered in the record

  1. Aleph Alpha releases open-weight model Kolibri

    On 3 October 2026 Aleph Alpha released Kolibri, an English-German mixture-of-experts Transformer with 78.1B total parameters and 3.46B active, under Apache 2.0 on Hugging Face. Context reaches 1M tokens, and Merlin-Arthur withholds answers the context does not support.

  2. Alibaba announces Qwen3.8-Max, its largest model to date

    Alibaba announced Qwen3.8-Max on 3 August 2026 with 2.4 trillion parameters, about 95 billion active at a time through a sparse mixture of experts, a context window up to 1 million tokens, and native text, image and video input. Weights were scheduled to follow a week later.

  3. DeepSeek publishes its V4 model series under an MIT license

    DeepSeek published a preview of its V4 series on 26 April 2026, comprising V4-Pro at 1.6 trillion total parameters with 49 billion active and V4-Flash at 284 billion with 13 billion active, released under the MIT license with checkpoints on Hugging Face.

  4. Alibaba splits Qwen into open and proprietary tiers

    Alibaba released Qwen3.6 in April 2026, a 35-billion-parameter mixture-of-experts model with 3 billion active parameters, under the Apache 2.0 license. It kept the more capable Qwen3.6-Plus and Qwen3.5-Omni proprietary, reachable only through Qwen’s apps and Alibaba Cloud.

  5. Mistral releases Small 4 and joins Nvidia’s Nemotron Coalition

    Mistral AI released Mistral Small 4 on 16 March 2026, a 119-billion-parameter mixture-of-experts model with 6 billion active parameters, a 256,000-token context window and an Apache 2.0 license. It joined Nvidia’s Nemotron Coalition the same day.

Every term in the glossary