Skip to content

Towards a Unified View of Preference Learning for Large Language Models: A Survey

ArXiv: 2409.02795

🎯 Pitch

This survey proposes a unified framework that decomposes preference learning into four interchangeable components—Model, Data, Feedback, and Algorithm—and shows how RL-style (e.g., PPO/RLHF) and SFT-style (e.g., DPO) methods fit a common gradient view. By clarifying the relationships between on-/off-policy data, feedback sources (humans, reward models, LLM judges), and optimization recipes, it makes it far easier to compare, combine, and scale alignment strategies—helping practitioners choose high-leverage components to reliably align LLM behavior before deployment.

1. Executive Summary (2-3 sentences)

This survey organizes “preference learning” methods for aligning LLM outputs with human preferences into a single unified framework that separates Model, Data, Feedback, and Algorithm (Section 3; Figure 5). Its main contribution is a common lens that connects historically separated families—e.g., RL-based PPO/RLHF and SFT-style DPO—by expressing them through a shared gradient form (Eq. (1)) and by decoupling “online/offline” from the optimization algorithm (Section 3; Appendix A). This matters because it clarifies which parts of an alignment recipe are interchangeable, making it easier to compare methods and to mix-and-match components (Introduction; Figure 2).

2. Context and Motivation

  • What specific problem or gap does this paper address?
  • The paper targets fragmentation in the alignment/preference-learning literature: methods are often grouped as “RL-based” (e.g., RLHF with PPO) vs “SFT-based” (e.g., DPO), which can obscure their shared structure and makes relationships across methods under-explored (Introduction).

  • It also notes confusion around “on-policy/off-policy” and “online/offline” terminology and argues these axes should be separated from algorithm choice (Section 3; Appendix A).

  • Why is this problem important?

  • Preference alignment is presented as a necessary step before deploying LLMs, to reduce unwanted behaviors like offensive/toxic/misleading outputs (Introduction; Figure 1 example).

  • The paper emphasizes that alignment often uses relatively small data to produce meaningful behavioral changes, so choosing the right components (data/feedback/algorithm) is high leverage (Abstract; Introduction).

  • What prior approaches existed, and where do they fall short (as framed here)?

  • Prior surveys and taxonomies often split the space into:
    • RL-based methods (e.g., RLHF + learned reward model + online RL such as PPO), versus
    • SFT-based methods (e.g., preference optimization using offline preference pairs, such as DPO) (Introduction; also referenced as a “traditional categorization”).

  • The paper argues this split can create an unnecessary conceptual barrier: the “core objective” and “optimization signal” are closely related across both categories (Introduction; Section 3).

  • How does this paper position itself relative to existing work?

  • It positions itself as providing:
    • A unified mathematical handle on gradients across methods (Eq. (1), Section 3),
    • A component decomposition (Model/Data/Feedback/Algorithm) with a process overview (Figure 5; Algorithm 1),
    • A taxonomy spanning data collection, feedback types, optimization families, and evaluation styles (Figure 2),
    • Concrete “working examples” diagrams of representative pipelines (Figures 3 and 4).

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

  • The “system” here is a conceptual framework for preference learning: a way to describe how an LLM is updated using preference-related signals.
  • It solves the problem of comparing and connecting alignment methods by decomposing each method into (1) where data comes from, (2) how “preference” is produced, and (3) how that signal enters a training (or decoding-time) algorithm (Section 3; Figures 2 and 5).

3.2 Big-picture architecture (diagram in words)

A box-and-arrows view matching Figure 5:

  1. Preference Data: collect (x, y) candidates (and sometimes multiple candidates per x) either on-policy or off-policy (Section 4).
  2. Environment / Feedback Generator: produce a preference signal r or ordering (human labels, rules, reward models, pairwise scorers, or LLM judges) (Section 5; Figure 6).
  3. Preference Optimization Algorithm: update the policy model π_θ (or adjust prompts/decoding without training) using point-wise, pair-wise, list-wise, or training-free methods (Section 6; Figure 2).
  4. Evaluation: measure alignment and capability using rule-based metrics and/or LLM-based judging (Section 7).

3.3 Roadmap for the deep dive

  • I first explain the paper’s formal definition of preference learning and the shared gradient view (Section 2; Section 3; Eq. (1)).
  • Then I detail the four components in the unified view:
  • Data (Section 4),
  • Feedback (Section 5),
  • Algorithms (Section 6),
  • Evaluation (Section 7).
  • Finally, I connect those components back to the paper’s claimed benefits (unification + mix-and-match) and identify limitations/trade-offs (Section 8; Appendix A/B).

3.4 Detailed, sentence-based technical breakdown

This is a survey + unifying framework paper, and its core idea is to represent most preference-learning variants as different ways of producing a training gradient from (data, feedback) and then applying an optimization rule (Section 3; Eq. (1); Figure 5).

3.4.1 Problem formulation and scope

  • The paper defines preference learning for an LLM policy π_θ as producing a new policy π_θ' that better aligns with a (conceptual) distribution of human preferences P(x, y) over prompts x and model outputs y (Section 2).
  • It explicitly scopes itself to textual preference alignment and excludes other alignment topics (the paper lists hallucination, multi-modal alignment, instruction tuning as out of scope) (Section 2).
  • It treats “alignment” as the broader behavioral alignment problem (creating an agent that behaves as humans want) and preference learning as one category of methods to achieve that (Section 2).

3.4.2 The unifying mathematical lens: a shared gradient template

  • The paper proposes that many RL-style and SFT-style alignment methods can be written in a shared gradient form (Section 3; Eq. (1)):

[ \nabla_\theta = \mathbb{E}{(q,o)\sim D}\Bigg[\frac{1}{|o|}\sum)\Bigg] ] (Eq. (1))}^{|o|}\delta_A(r,q,o,t)\ \nabla_\theta \log \pi_\theta(o_t\mid q,o_{<t

Here is what each part means in plain language (using the paper’s notation and descriptions in Section 3):

  • D is the data source containing questions/prompts q and outputs o.
  • δ_A(r,q,o,t) is a gradient coefficient: it determines direction and step size of the update, and depends on:
    • the algorithm A,
    • the feedback r (broadly defined),
    • and the data instance/time step t.

  • The model update still looks like “increase log-prob of certain tokens,” but the coefficient tells you which sequences/tokens should be upweighted or downweighted based on preference.

  • A key conceptual move is the paper’s definition of feedback: anything from the “environment” that influences this gradient coefficient—scalar rewards, preference labels, rankings, correctness indicators, etc. (Section 3; Section 5).

3.4.3 Decoupling “algorithm choice” from online/offline

  • The paper argues “online vs offline” depends on whether feedback can be obtained in real time for newly sampled model outputs, not on whether you use an RL or SFT objective (Section 3; Appendix A).
  • Concretely:
  • In an online setting, you sample responses from the current π_θ and query an environment (human, RM, rules, etc.) to get feedback on-the-fly (Algorithm 1, lines 5–8).
  • In an offline setting, you train from a pre-built dataset that already contains preference signals (Algorithm 1, lines 9–11).
  • The paper highlights that even DPO-style objectives could be used online if you can label pairwise preferences on newly generated data in real time (Section 3).

3.4.4 System/data pipeline diagram in words (explicit first→second→third flow)

A concrete flow consistent with Figure 5 and Algorithm 1:

  1. Initialize the model π_θ that you want to align, and optionally set a reference model π_ref ← π_θ if the algorithm needs it (Algorithm 1, lines 1–3; Section 6.2 discusses π_ref in DPO).
  2. Obtain a batch of prompts:
  3. Either sample from unlabeled queries Q (online) (Algorithm 1, line 6),
  4. Or load a batch from an offline dataset D (offline) (Algorithm 1, line 10).
  5. Generate candidate outputs y (or multiple candidates) with the current policy π_θ (Section 4; Figures 3–4 illustrate sampling multiple candidates).
  6. Get feedback from an environment aligned with human preference:
  7. This may produce a scalar r, a label (“good/bad”), a pairwise relation (y_i > y_j), or a ranking over a list (Section 5; Figure 6).
  8. Run a preference optimization algorithm A that converts (batch data + feedback) into parameter updates for π_θ (Algorithm 1, line 12; Section 6).
  9. Evaluate the resulting aligned model using rule-based metrics or LLM-based judging (Section 7; Figure 5).

3.4.5 Component 1 — Preference Data (Section 4)

  • The paper uses a simple notation for preference data as (x, y, r) where r is some preference signal (label/score/etc.) (Section 4).
  • It divides data collection into:
  • On-policy: data is sampled from the current model π^t_θ during training (Section 4.1).
  • Off-policy: data is collected independently of the current training step/model, including:
    • human preference datasets, and
    • LLM-generated preference datasets (Section 4.2).

On-policy decoding strategies (Section 4.1): - For sampling diverse outputs, it lists common decoding methods such as Top-K/nucleus sampling and beam search (Section 4.1). - For multi-step problems, it discusses MCTS (Monte Carlo Tree Search) as a way to generate better and more diverse candidate solution trajectories and sometimes obtain step-level labels (Section 4.1).

Off-policy sources (Section 4.2): - From humans: examples include WebGPT comparisons (20K), OpenAI human preferences from TL;DR, HH-RLHF (170K chats with pairwise preferences), and SHP (385K preferences) (Section 4.2). - From LLMs: examples include RLAIF (AI-labeled preferences), Open-Hermes-Preferences (~1M), UltraFeedback (GPT-4-based), UltraChat (multi-turn instructional conversations) (Section 4.2).

3.4.6 Component 2 — Feedback mechanisms (Section 5)

The paper splits feedback into direct vs model-based (Section 5; Figure 6).

A) Direct feedback (no trained reward model required) (Section 5.1) - Labeled datasets: offline preference labels annotated by humans can be used directly (Section 5.1). - Hand-designed rules: task-specific rules turn outcomes into feedback signals: - Math: correctness checks can yield binary reward like r = I(c) (Section 5.1). - Theorem proving: proof assistants/tools can provide feedback (Section 5.1). - Translation: a reference-free QE model can generate preference signals used in CPO (Section 5.1). - Code: unit tests and heuristics can rank/score code generations; these scores then affect training loss (Section 5.1). - Summarization: user edits can act as interactive feedback (Section 5.1).

B) Model-based feedback (Section 5.2) - Reward models (RM): - A Bradley–Terry style preference model defines the probability that output y1 is preferred over y2 given x using exponentiated rewards (Eq. (2)) and trains via a logistic loss on chosen vs rejected outputs (Eq. (3)) (Section 5.2.1). - The paper also describes point-wise binary classifier reward models for tasks where “correct vs incorrect” can be labeled directly (Eq. (4)) (Section 5.2.1). - It surveys directions to improve RM training: - better synthetic preference data generation (e.g., regularized Best-of-N / West-of-N), - ensembling / MoE-style modeling for uncertainty and overoptimization control, - fine-grained / process supervision vs outcome-only supervision, and - prior constraints to stabilize reward scaling (Section 5.2.1). - Pair-wise scoring models: - These are lightweight models specialized for pairwise comparisons, argued to be more consistent than assigning absolute scores; limitations include not producing global scores and limited candidate set size (Section 5.2.2). - LLM-as-a-Judge: - Larger LLMs can be prompted with scoring rubrics to judge outputs and provide rewards or labels; the paper notes potential errors/biases and mentions meta-rewarding/self-improvement schemes as ways to refine judging (Section 5.2.3).

3.4.7 Component 3 — Algorithms (Section 6)

The paper’s primary algorithm taxonomy is based on how many samples are needed to compute the gradient coefficient in Eq. (1): point-wise, pair-wise, list-wise, plus training-free (Section 6; Figure 2).

A) Point-wise methods (Section 6.1) - Rejection sampling fine-tuning: - Filter/select high-quality (x, y+) and then do standard maximum-likelihood training on selected outputs (Eq. (5)) (Section 6.1). - Limitation: it discards low-reward data and therefore “doesn’t learn from non-preferred data” (Section 6.1). - PPO-style RLHF: - The paper presents a reward-maximization objective with a KL penalty to a reference model π_ref, scaled by a coefficient β (Eq. (6)) (Section 6.1). - It highlights drawbacks: computational cost, sample inefficiency, training instability, and hyperparameter sensitivity (Section 6.1). - ReMax: - A REINFORCE-like approach that uses a baseline reward from greedy decoding to avoid training a critic model; the paper notes a classification nuance (it can be seen as pairwise “from the gradient coefficient perspective”) (Section 6.1; Appendix B). - KTO: - Uses only a binary label (preferred or not) and is presented as stable and hyperparameter-light, motivated by prospect theory framing (Section 6.1).

B) Pair-wise contrasts (Section 6.2) - Motivation: point-wise methods may over-focus on positives or be harder to optimize; pairwise methods explicitly contrast preferred vs non-preferred candidates (Section 6.2). - DPO: - It connects RLHF and reward modeling by relating reward to a log-ratio between the learned policy and a reference policy (Eq. (7)) and yields a logistic objective on the difference of log-ratios for chosen vs rejected outputs (Eq. (8)) (Section 6.2). - The paper also lists a family of DPO-like variants and summarizes several in Table 1 (e.g., IPO, f-DPO, EXO, DPO-positive, ORPO, SimPO) with notes about key hyperparameters (β, τ, λ, γ, etc.) (Section 6.2; Table 1). - IPO: - Presented as addressing overfitting by replacing an unbounded mapping used in DPO-style scoring with an identity mapping and imposing an upper bound effect via a margin parameter τ (Section 6.2; Table 1). - Pipeline variants / online variants: - The paper describes methods that (i) combine SFT + preference optimization into one stage (e.g., ORPO), and (ii) update the DPO reference model dynamically for online-style training (Section 6.2).

C) List-wise contrasts (Section 6.3) - These extend pairwise comparisons to lists of candidates for a prompt. - The paper discusses: - RRHF: expands to lists but still forms multiple pairs (Section 6.3). - PRO: recursively applies list-wise contrasts (Section 6.3; Table 1 includes a PRO formulation using an external reward model r*). - Calibration/reweighting approaches to reduce bias and overfitting in listwise estimation (Section 6.3). - GRPO: samples a group of responses, scores them, normalizes rewards within-group, and uses the normalized reward as an advantage—removing the need for a critic model compared to PPO (Section 6.3). Appendix B clarifies the classification nuance: it is listwise in how it computes the baseline/advantage but pointwise in how it applies the per-sample loss after that.

D) Training-free alignment (Section 6.4) These do not update model parameters; they adjust either inputs or decoding outputs.

  • Input optimization:
  • Use system prompts, in-context examples, retrieval of norms, or prompt rewriting to induce aligned outputs (Section 6.4.1).
  • Examples discussed include URIAL-style prompting based on token distribution shifts, retrieval of norms, and BPO prompt rewriting (Section 6.4.1).
  • Output optimization:
  • Add a post-hoc “aligner” rewrite model, manipulate logits during decoding, or do search/backtracking when harmful content is detected (Section 6.4.2).
  • Examples include an alignment module (Aligner), logits manipulation approaches, rewindable decoding (RAIN), and in-context selection schemes like ICDPO (Section 6.4.2).

3.4.8 Worked micro-example (illustrative, not a paper-reported result)

To make Eq. (8) tangible, consider a single prompt x with a chosen response y+ and rejected response y-. Define:

  • a = log ( π(y+|x) / π_ref(y+|x) )
  • b = log ( π(y-|x) / π_ref(y-|x) )
  • DPO uses loss: L = -log σ( β(a - b) ) (Eq. (8))

If β = 1, a = 0.2, b = -0.1, then: - a - b = 0.3, so σ(0.3) ≈ 0.574 - L ≈ -log(0.574) ≈ 0.555

If the model instead assigns relatively higher ratio to the rejected answer (say a = 0.0, b = 0.4): - a - b = -0.4, σ(-0.4) ≈ 0.401 - L ≈ -log(0.401) ≈ 0.915 (worse)

So minimizing the DPO loss pushes the model to increase the chosen-vs-rejected log-ratio gap (the exact mechanics are in Eq. (8); this numeric example is only to illustrate the direction).

3.4.9 Required element: configurations and hyperparameters

  • This survey does not specify a single training run (model size, optimizer settings, batch sizes, tokens, hardware) because it is not introducing one new trained model; instead it catalogs methods and sometimes mentions hyperparameters as part of algorithm definitions (e.g., β in DPO/PPO objectives, τ in IPO, Îť in ORPO/DPO-positive, Îł in SimPO; Eq. (6); Table 1).
  • Where the paper does provide hyperparameter roles, it is in the objective definitions:
  • β as the KL/divergence or scaling coefficient (Eq. (6), Eq. (8), Table 1),
  • algorithm-specific margin/weight parameters (τ, Îť, Îł) (Table 1).
  • Any additional training details (optimizer, LR schedule, batch size, context window, tokens, compute, hardware) are not present in the provided content, and it would be speculative to invent them.

4. Key Insights and Innovations

  • (1) A four-component decomposition: Model, Data, Feedback, Algorithm
  • The paper’s central organizational contribution is decomposing preference learning into these components (Section 3; Figure 5), rather than categorizing primarily by RL vs SFT or online vs offline.

  • Significance: this decomposition clarifies which parts of a pipeline can change independently (e.g., same algorithm with different feedback types; same feedback type with different data regimes).

  • (2) Unified gradient perspective across RL-style and SFT-style alignment

  • Eq. (1) presents a common gradient template where differences across methods concentrate in the gradient coefficient δ_A(…), which is driven by feedback form and algorithm design (Section 3; Eq. (1)).

  • Significance: it reframes “reward vs preference labels vs rankings” as different ways to shape δ, making RLHF, DPO-style training, and filtering/ranking methods more comparable mechanistically.

  • (3) Decoupling algorithm families from online/offline settings

  • The paper argues online/offline is about whether feedback is available in real time, and that algorithms like DPO need not be inherently offline (Section 3; Algorithm 1; Appendix A).

  • Significance: this opens conceptual room for hybrid pipelines (e.g., online data with pairwise labeling; offline PPO-style updates with stored rewards as shown in Figure 3).

  • (4) A taxonomy that spans the full lifecycle, including evaluation

  • Figure 2 maps methods across data collection, feedback, optimization type, and evaluation (Figure 2; Sections 4–7).
  • Significance: it encourages thinking about evaluation bias and feedback reliability as first-class design constraints, not afterthoughts (Section 7.2.3).

5. Experimental Analysis

  • Evaluation methodology described (but not newly executed)
  • Because this is a survey, it primarily summarizes evaluation approaches rather than reporting new experimental results.

  • It describes two main evaluation families (Section 7):

    • Rule-based evaluation using ground-truth-based metrics like Accuracy/F1/Exact Match/ROUGE and benchmark suites spanning factual knowledge, math, reasoning, closed-book QA, and coding (Section 7.1).
    • LLM-based evaluation using an LLM as judge (pairwise comparison, single answer grading, reference-guided grading) (Section 7.2.1).

  • Baselines and quantitative results

  • The provided content does not include new benchmark tables of model performance, win rates, or improvements attributable to the survey’s framework (i.e., there are no new experiments to summarize with numbers).

  • The closest “quantitative” content is descriptive statistics about datasets (e.g., WebGPT 20K comparisons; HH-RLHF 170K chats; SHP 385K) (Section 4.2) and the presence of algorithmic formulas (Eq. (5)–(8), Table 1).

  • Do experiments convincingly support claims?

  • Since the paper’s core claims are about conceptual unification and taxonomy, the “support” is mostly the coherence of:
    • the unified gradient form (Eq. (1)),
    • the process formalization (Algorithm 1),
    • the taxonomy coverage (Figure 2),
    • and worked pipeline diagrams (Figures 3–4).

  • There is no empirical validation in the provided content that shows, for example, that using this decomposition leads to better aligned models; that is outside the survey’s scope.

  • Ablations / failure cases / robustness checks

  • Not applicable as new experiments are not presented.
  • The paper does, however, discuss known pitfalls/biases in evaluation—e.g., position bias and verbosity bias in LLM judges (Section 7.2.3)—which functions as a qualitative robustness discussion.

6. Limitations and Trade-offs

  • Survey scope limitations

  • The paper explicitly focuses on textual preference alignment and excludes other alignment topics like hallucination, multimodal alignment, and instruction tuning (Section 2). This means the unified view may not directly cover those adjacent areas.

  • Unification via Eq. (1) may hide important differences

  • The gradient template (Eq. (1)) centralizes differences into δ_A(…), which is useful, but it can also gloss over practical distinctions such as:
    • stability properties of different losses,
    • sensitivity to reward-model errors,
    • and implicit regularization effects from reference models.

  • The paper partially acknowledges such nuances via discussions like ReMax/GRPO classification ambiguity (Appendix B), suggesting that categorization is sometimes perspective-dependent.

  • Feedback reliability is a bottleneck

  • The paper emphasizes that feedback (human, rules, RM, LLM judges) is foundational (Section 5), and later flags “reliable feedback and scalable oversight” as a major open challenge (Section 8).

  • Trade-off: scalable feedback (e.g., LLM-as-a-judge) may introduce bias/errors (Section 5.2.3; Section 7.2.3), while high-quality human feedback is expensive (Section 4.2).

  • Evaluation can be systematically biased

  • LLM-based evaluators can have position bias, verbosity preference, similarity bias, and weaknesses on math/reasoning-like domains (Section 7.2.3).

  • This creates a trade-off where the “fast” evaluation tools may mis-rank models or incentivize superficial improvements.

  • Lack of prescriptive guidance on “which variant wins when”

  • The survey concludes that algorithms share core objectives but vary across scenarios, and it explicitly leaves “which variants perform better in specific contexts” as future work (Conclusion).
  • Practically, this means the framework helps organize choices but does not, by itself, decide them.

7. Implications and Future Directions

  • How this work changes the landscape
  • The main impact is conceptual: it encourages the field to reason about preference learning as an interchangeable pipeline—data + feedback + algorithm—rather than as siloed families (RLHF vs DPO) (Section 3; Figure 5).

  • It also legitimizes hybridization: e.g., using DPO-like objectives with online feedback, or using PPO-like updates with offline stored rewards (Section 3; Figure 3 examples).

  • Follow-up research it enables or suggests (from Section 8)

  • Better quality and more diverse preference data:
    • The paper argues performance depends strongly on data quality/diversity, suggesting more work on synthetic data quality control and more diverse sampling techniques (Section 8).
  • Reliable feedback and scalable oversight:
    • Extending high-reliability feedback (compilers, proof assistants) beyond narrow domains, and developing methods for cases where humans cannot reliably evaluate outputs (Section 8).
  • Advanced algorithms for preference learning:
    • Algorithms should better approach an upper bound determined by data/feedback, be robust to imperfect signals, and be efficient/stable at scale (Section 8).

  • More comprehensive evaluation:

    • The paper calls for evaluation methods that are more diverse, less biased, and less costly, given limitations of both rule-based metrics (for open-ended tasks) and LLM judges (biases) (Section 8; Section 7).

  • Practical applications / downstream use cases

  • The survey’s component view directly maps to real deployment choices:

    • If you have domain verifiers (unit tests, proof tools), you can use direct feedback loops (Section 5.1).
    • If you need general-purpose alignment but lack human labels, you might rely on model-based feedback (RMs or LLM judges) (Section 5.2).
    • If fine-tuning is costly or impossible, you can apply training-free alignment via prompt/decoding optimization (Section 6.4).

  • Repro/Integration Guidance (based on the paper’s framework)

  • Using the paper’s four components (Figure 5), a practical selection heuristic is:
    • Prefer direct feedback when objective rules exist (math correctness, unit tests), because it avoids training an RM and can be more reliable in-domain (Section 5.1).
    • Prefer pairwise/listwise algorithms when you can obtain comparisons/rankings and want the model to learn contrasts between good and bad outputs (Section 6.2–6.3).
    • Prefer training-free methods when you cannot update model weights or need lightweight deployment-time control (Section 6.4).
  • The paper also cautions that evaluation choice matters: if you use LLM-as-a-judge for evaluation, account for known biases like position and verbosity effects (Section 7.2.3), and consider meta-evaluation resources (Section 7.2.3).