Distillation
Training a small model to imitate a large one, so it answers nearly as well at a fraction of the running cost.
Instead of training a small model from scratch, a large “teacher” model generates answers and the small “student” is trained to reproduce them. The student ends up cheaper to run and easier to distribute, though it rarely matches the teacher on the hardest questions. Distillation became a central cost story in 2024 and 2025, as labs raced to make their best models affordable to serve.
Where it mattered in the record
-
Jensen Huang defends AI distillation as competition
On 28 September 2026 Nvidia CEO Jensen Huang told CNBC’s Squawk Box that training AI models on rivals’ outputs is ‘not robbery’: ‘That’s called competition.’ ‘You’re allowed to test somebody else’s products all you want,’ he said. Nvidia’s products get stripped ‘down to bones.’
-
Anthropic publishes September threat-intelligence report
On 10 September 2026 Anthropic published a 154-page threat-intelligence report on malicious uses of Claude it stopped between December 2025 and August 2026, across cyber operations, surveillance, fraud, weapons work, and illicit model distillation.