In RLHF, what is Data Collection used for?

In RLHF, what is Data Collection used for?

• A. Gathers unlabeled data from the web to train the base model.
• B. Gathers human-preference examples to train a reward model.
• C. Collects system performance metrics to replace reward learning.
• D. Only used for post-deployment monitoring and not training.

Answer: (B)

In RLHF, data collection is about gathering human-preference data to train a reward model. Humans review and rank or choose between different model outputs for given prompts, producing pairs or rankings that reflect which responses are preferred. This information teaches a reward model to predict human judgments, effectively encoding what people value in the model’s behavior. That reward model then provides a scalar signal used to guide the policy through reinforcement learning, steering the base model toward outputs aligned with human preferences. This focus on human judgments for the reward model is what makes data collection in RLHF specifically about obtaining those preferences, rather than collecting unlabeled data for base training or relying on post-deployment metrics.

In RLHF, data collection is about gathering human-preference data to train a reward model. Humans review and rank or choose between different model outputs for given prompts, producing pairs or rankings that reflect which responses are preferred. This information teaches a reward model to predict human judgments, effectively encoding what people value in the model’s behavior. That reward model then provides a scalar signal used to guide the policy through reinforcement learning, steering the base model toward outputs aligned with human preferences. This focus on human judgments for the reward model is what makes data collection in RLHF specifically about obtaining those preferences, rather than collecting unlabeled data for base training or relying on post-deployment metrics.