Which sequence correctly describes the RLHF process?

Which sequence correctly describes the RLHF process?

• A. Data collection; Build reward model; Supervised fine-tuning; Optimize language model with the reward-based model.
• B. Data collection; Supervised fine-tuning; Build reward model; Optimize language model with the reward-based model.
• C. Supervised fine-tuning; Data collection; Build reward model; Optimize language model.
• D. Data collection; Supervised fine-tuning; Build reward model; Optimize language model with the reward-based model.

Answer: (D)

RLHF trains a model to align with human preferences through a four-step flow: data collection, supervised fine-tuning, reward model construction, and policy optimization using that reward signal. Data collection provides human demonstrations and preference data that show what good behavior looks like and how outputs should be judged. Supervised fine-tuning then uses those demonstrations to teach the base language model to imitate helpful and safe responses, giving it a solid starting point. Building the reward model uses human judgments—often comparing outputs—to learn a scoring function that can distinguish better from worse responses. Finally, optimizing the language model with the reward-based model uses reinforcement learning to adjust the model so its outputs maximize the reward, i.e., align more closely with human preferences. Other sequences misplace these steps in a way that either scores outputs without a capable generator or tries to train the reward before there’s a reliable model to evaluate, which is why this order best reflects the standard RLHF workflow.

RLHF trains a model to align with human preferences through a four-step flow: data collection, supervised fine-tuning, reward model construction, and policy optimization using that reward signal. Data collection provides human demonstrations and preference data that show what good behavior looks like and how outputs should be judged. Supervised fine-tuning then uses those demonstrations to teach the base language model to imitate helpful and safe responses, giving it a solid starting point. Building the reward model uses human judgments—often comparing outputs—to learn a scoring function that can distinguish better from worse responses. Finally, optimizing the language model with the reward-based model uses reinforcement learning to adjust the model so its outputs maximize the reward, i.e., align more closely with human preferences. Other sequences misplace these steps in a way that either scores outputs without a capable generator or tries to train the reward before there’s a reliable model to evaluate, which is why this order best reflects the standard RLHF workflow.