RLHF
Reinforcement learning from human feedback: tuning a chatbot with people’s rankings of its answers.
After the main training, the lab has people rank pairs of the model’s answers by quality, trains a scoring model on those rankings, and then tunes the chatbot to score well. That is reinforcement learning from human feedback, RLHF, and it is what turned raw language models into assistants that follow instructions and mind their manners. It has limits: it optimizes for answers people rate highly, which is not the same as answers that are true.