Direct Preference Optimization: Your Language Model is Secretly a Reward Model · HackerPapers discussions

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by XTXinverseXTY

Thu, 24 Sep 2026 22:38:42 GMT

Hard to separate the memorability of the abstract from the underlying material. IMO the most memorable papers contain some unexpected simplification: a complicated problem turns out to reduce to something much simpler [0], perhaps for a counterintuitive reason [1]. So a memorable abstract should advertise that the paper contains some cute little trick. This may not help you actually publish though [0]: Attack the RLHF problem with a simple classification loss: https://arxiv.org/abs/2305.18290 [1]: Frustratingly Easy Meta-Embeddings: https://arxiv.org/abs/1804.05262

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by weitendorf

Sun, 07 Jun 2026 07:26:53 GMT

You should note that Claude Design is most likely a DPO->PPO->Actor-Critic bootstrap play: https://arxiv.org/abs/2305.18290 / https://en.wikipedia.org/wiki/Proximal_policy_optimization / https://spinningup.openai.com/en/latest/algorithms/sac.html It's much harder to RL out design taste because it's not self-grounding, and human labelers have no real skin in the game, so this (having a human with a vested outcome in the process directing a model's work) is the best way to get LLMs better at design/"taste"/aesthetic judgment themselves. We were working on the same thing 7 months ago and then I…

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by thinkzilla

Mon, 18 Aug 2025 05:59:27 GMT

While the post uses DPO to illustrate RL and RLHF, in fact DPO is an alternative to RLHF that does not use RL. See the abstract of the DPO paper https://arxiv.org/abs/2305.18290, and Figure 1 in the paper: "DPO optimizes for human preferences while avoiding reinforcement learning". The confusion is understandable. The definition of RL in the Sutton/Barto book extends over two chapters iirc, and after reading it I did not see how it differed from other learning methods. Studying some of the academic papers cleared things up.

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by column

Thu, 25 Jan 2024 15:31:53 GMT

I kind of disagree. It's not "user friendly" but it is very descriptive. They are codenames afterall. Take "dolphin-2.6-mistral-7b-dpo-laser" for instance : with a little LLM background knowledge, just from the name you know it is a 7 billion parameters model based on Mistral, with a filtered dataset to remove alignment and bias (dolphin), version 2.6 and using the techniques described in the Direct Preference Optimization (https://arxiv.org/pdf/2305.18290.pdf) and Laser (https://arxiv.org/pdf/2312.13558.pdf) papers to improve its output.

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — HN submission

Fri, 12 Jan 2024 19:22:07 GMT

Direct Preference Optimization: Your Language Model Is a Reward Model

1 point; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — HN submission

Thu, 11 Jan 2024 19:01:28 GMT

Direct Preference Optimization: Your Language Model Is a Reward Model

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by thatguysaguy

Fri, 15 Dec 2023 21:24:02 GMT

DPO should be listed as well: https://arxiv.org/abs/2305.18290 It's extremely zeitgeisty atm

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — HN submission

Tue, 05 Sep 2023 15:58:50 GMT

Direct Preference Optimization: Your Language Model Is Secretly A Reward Model

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by wmwmwm

Wed, 21 Jun 2023 19:49:14 GMT

Does anyone have any insight into why reinforcement learning is (maybe) required/historically favoured? There was an interesting paper recently suggesting that you can use a preference learning objective directly and get a similar/better result without the RL machinery - but I lack the right intuition to know whether RLHF offers some additional magic! Here’s the “ Direct Preference Optimization ” paper: https://arxiv.org/abs/2305.18290

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — HN submission

Mon, 05 Jun 2023 02:04:51 GMT

Direct Preference Optimization: Your Language Model Is a Reward Model

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — mentioned by fnands

Wed, 31 May 2023 15:17:23 GMT

Have you had a chance to look at the Direct Preference Optimization [1] pre-print? It seems like it might help get around RLHF which (as far as I can tell) is the really hard/expensive part of training the best of these models. They said they will release the code soon, so I guess we will found out soon enough. [1] https://arxiv.org/abs/2305.18290

Paper and discussion history. A mention is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — HN submission

Wed, 31 May 2023 13:06:51 GMT

Direct Preference Optimization: Your Language Model Is a Reward Model

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model — HN submission

Tue, 30 May 2023 02:00:17 GMT

RLFH Alternative Direct Preference Optimization

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.