Transformers

2 entries between June 2018 and October 2018.

Every entry

  1. Google publishes BERT, a bidirectional language model

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova published BERT on 11 October 2018, a transformer that pretrains representations of text by reading in both directions, then fine-tuned for particular tasks. Google released the code and weights the following month.

  2. OpenAI publishes the GPT-1 paper on pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever published “Improving Language Understanding by Generative Pre-Training” on 11 June 2018, describing a transformer pretrained on unlabeled text and then fine-tuned for particular tasks.