Fine-Tuning Language Models from Human Preferences · HackerPapers discussions

Fine-Tuning Language Models from Human Preferences — mentioned by icepush

Wed, 09 Sep 2026 08:39:30 GMT

Imagine one of the recent frontier models with a flipped sign (cf §4.4 of https://arxiv.org/pdf/1909.08593)

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by ben_w

Sat, 16 Aug 2025 10:53:30 GMT

That started years ago. Exactly when depends what you count, but part of the path leading to ChatGPT was getting a bunch of dumb systems to work together to train a smarter system: https://arxiv.org/pdf/1909.08593 fig 1, and also section 4.4 for what can go wrong

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by crackalamoo

Fri, 31 Jan 2025 14:14:13 GMT

As far as I can tell, OpenAI really did invent RLHF in its current form. https://arxiv.org/pdf/1909.08593

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by ben_w

Wed, 03 Apr 2024 10:04:42 GMT

> Lots of developers here, but also in the media, believe GPT is the precursor for AGI. Which is ridiculous; it's just an incredible token predictor. Its language capabilities might be useful, but AGI won't be possible without a self-correcting model that can actually reason. Even Altman hinted at that. I think part of this is that all three letters of the initialism mean different things to different people. To me? The domain is broad rather than narrow, so it's got the "G" of "AGI". Definitely agree it is not a complete solution, and thus that reasonable people will say "no Ben this isn't…

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by swyx

Wed, 06 Sep 2023 18:31:13 GMT

i highlighted this last night which seems to be making the rounds - https://twitter.com/swyx/status/1699369076529971545 Alec's CV (https://www.linkedin.com/in/alecradford/) seems to be: - 2011-2016 BSc Eng from Olin College - 2013+ started a data/AI consultancy as a sophomore that turned into a vague startup/product? - 2015 first paper on GANs coauthored with soumith https://arxiv.org/pdf/1511.06434.pdf%C3 - 2016 first GAN paper under openai email https://proceedings.neurips.cc/paper_files/paper/2016/file/8... - 2017 the generative reviews paper mentioend in the Wired article, with Ilya…

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by mistercow

Fri, 01 Sep 2023 13:12:12 GMT

My understanding is that there’s no RLHF model sitting on top of ChatGPT. There’s a separate reward model during training, but that reward model is just used to fine-tune the decoder model. This is sort of confusing in the literature, because they will switch from calling it a “language model” to a “policy”, once they’ve used PPO, but if you trace the citations back to [1], you can see that the policy is initialized with a pretrained language model (in that case, GPT-2). What I don’t know is how they’re applying RLHF after user fine-tuning. Are they redoing the PPO with the original reward…

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by jefftk

Tue, 14 Mar 2023 12:51:01 GMT

Reinforcement learning in general didn't come out of AI alignment work, but RHLF in particular did. The initial idea and paper [1] were from AI alignment folks, as was most of the later development [2][3][4][5]. Overview: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoug... [1] Deep Reinforcement Learning from Human Preferences https://arxiv.org/abs/1706.03741 [2] Fine-Tuning Language Models from Human Preferences https://arxiv.org/abs/1909.08593 [3] Learning to summarize from human feedback https://arxiv.org/abs/2009.01325 [4] Recursively Summarizing Books with Human Feedback…

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Language Models from Human Preferences — mentioned by gwern

Sat, 21 Dec 2019 15:21:11 GMT

What's particularly nice is if you can plug in a classifier for things like esthetics based on human ratings, along the lines of 'learning from human preferences' https://arxiv.org/abs/1909.08593 but better - why spend the enormous effort running PPO to brute force the classifier to obtain desired text or image output, when you can just backprop through it and let the classifier itself tell you how exactly to improve the inputs? (I was writing up this idea as a proposal for improving preference learning at https://www.gwern.net/GPT-2-preference-learning#optimization... based on my experiences…

Paper and discussion history. A mention is not an endorsement.