How does RLHF work?

How does RLHF work?

• A. The model outputs A and B; a human says 'A is better'; the model gets signal to produce more A and less B
• B. The model is trained with labels only after the end of training
• C. The model randomly updates parameters without feedback
• D. The model learns by maximizing likelihood of training data without human input

Answer: (A)

Reinforcement learning from human feedback uses a human-provided signal to steer a model toward outputs that people prefer. The idea is to convert qualitative judgments into a reward signal the model can optimize against. In practice, humans compare different model outputs and indicate which is better. A separate reward model is often trained to predict these human preferences, creating a numeric score for how desirable an output is. The main model is then updated with reinforcement learning to maximize this reward, so it learns to produce responses that align with human preferences over time. This matches the described option: when a human flags one output as better, the model adjusts to favor that kind of output in the future, increasing the likelihood of producing A over B. Why the other ideas don’t fit: one option suggests training solely with end-of-training labels, which misses the ongoing feedback loop that shapes behavior during learning. Another implies random updates without feedback, which defeats the purpose of learning from preferences. The last suggests learning by maximizing the likelihood of training data without any human input, which describes standard supervised pretraining rather than the human-guided refinement characteristic of RLHF.

Reinforcement learning from human feedback uses a human-provided signal to steer a model toward outputs that people prefer. The idea is to convert qualitative judgments into a reward signal the model can optimize against. In practice, humans compare different model outputs and indicate which is better. A separate reward model is often trained to predict these human preferences, creating a numeric score for how desirable an output is. The main model is then updated with reinforcement learning to maximize this reward, so it learns to produce responses that align with human preferences over time.

This matches the described option: when a human flags one output as better, the model adjusts to favor that kind of output in the future, increasing the likelihood of producing A over B.

Why the other ideas don’t fit: one option suggests training solely with end-of-training labels, which misses the ongoing feedback loop that shapes behavior during learning. Another implies random updates without feedback, which defeats the purpose of learning from preferences. The last suggests learning by maximizing the likelihood of training data without any human input, which describes standard supervised pretraining rather than the human-guided refinement characteristic of RLHF.