A training method for improving AI answers with human preference data.
DPO is like taste-testing two cookies. Pick the winner, and the baker changes the recipe.
It is used after basic training to align a model. It makes answers fit people better, with less training fuss.
RLHF
DPO is a simpler path than RLHF, with no reward model detour.
Post-training
DPO is often used after pre-training to align model preferences.
Fine-tuning
DPO uses preference examples to keep fine-tuning model behavior.
Alignment
DPO makes model answers closer to human preferences.