OpenAI unveils GPT-2 and withholds the full model
OpenAI announced GPT-2 on 14 February 2019 with samples of coherent multi-paragraph text, and published only a 124-million-parameter version rather than the full 1.5-billion-parameter model, citing concerns about malicious use. It called the plan a staged release.
Why it mattered A major laboratory argued for the first time that a language model’s capability alone justified withholding it, opening a dispute over release norms that ran through the year.
The convention in machine-learning research was to publish the paper and the weights together. A model was an artifact of a scientific claim, and other laboratories were expected to be able to check it. OpenAI had followed that convention itself the previous June with the GPT-1 paper, which showed that a transformer pretrained on a large amount of ordinary text could then be fine-tuned to do particular language tasks well.
GPT-2 was the same recipe made larger. On 14 February 2019 OpenAI described a model with 1.5 billion parameters trained to predict the next word in text, and published samples in which it continued a human prompt for several paragraphs while holding its subject and register. Then it said it would not release the model. Citing “concerns about malicious applications of the technology,” the company published a 124-million-parameter version instead and described what it was doing as a staged release, with larger versions to follow if the risk looked manageable.
The reasoning was new. Earlier arguments for holding a result back had concerned data that was private or work that was dangerous by construction. This was a claim about a general-purpose text generator: that it might be used for disinformation, spam, or impersonation at a scale that made publication itself the risk. The claim was also hard to test, since nobody outside OpenAI could examine the model it was made about, and the argument ran through the year on reasoning rather than evidence.
Larger versions followed over the months that came after, with outside groups studying misuse in between. On 5 November OpenAI released the full model and its code, saying it had found no strong evidence of misuse of the versions already published. The episode supplied the vocabulary, staged release among the terms, for arguments about publishing capable models that have continued since.