Training language models to follow instructions with human feedback · HackerPapers discussions

Training language models to follow instructions with human feedback — mentioned by refulgentis

Sun, 20 Sep 2026 00:50:13 GMT

4th author on "Training language models to follow instructions with human feedback" https://arxiv.org/html/2203.02155v1. Pure lie?

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by CompleteSkeptic

Sat, 06 Jun 2026 01:25:39 GMT

I helped train some of the first "magic" models at OpenAI[1] and it was a wild ride. We were a pretty sane + skeptical team and we weren't totally convinced the models were as general as they seemed, but the query that convinced me (and later got included in the paper[2]) was "Why is it important to eat socks after meditating?" (something that almost certainly did not appear on the internet before). An interesting follow up would be when did you realize GenAI wasn't as good as you thought in that "oh shit" moment [1] co-author of InstructGPT/RLHF/ChatGPT [2] https://arxiv.org/pdf/2203.02155

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by godelski

Mon, 11 Aug 2025 01:11:29 GMT

Jack has a lot of really bad takes and frequently makes lots of mistakes. Honestly, I don't know why people take him seriously. I mean you can go read his blog post that's pinned where he argues that there's no new ideas and it is all data. He makes the argument that architecture doesn't matter, which is just so demonstrably false that it is laughable. He's a scale maximalist. I also expect an AI researcher from a top university to not make such wild mistakes > 3. RLHF: first proposed (to my knowledge) in the InstructGPT paper from OpenAI in 2022 I mean if you go read the instruct paper on…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by bbor

Thu, 20 Jun 2024 01:29:30 GMT

I personally find that summary of things to be way off the mark (for example, hopefully "the face" you reference isn't based on anything that appears in a browser window or in an ensemble of less than 100 agents!) but I'll try to speak to the "united" question instead. 1. The "Future of Life" institute is composed of lots of very serious people who recently helped get the EU "AI Act" passed this March, and they discuss the "myriad risks and harms AI presents" and "possibly catastrophic risks". https://newsletter.futureoflife.org/p/fli-newsletter-march-2... 2. Many researchers are leaving…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by throwaway4aday

Sat, 01 Jun 2024 23:35:12 GMT

> Usage data: ChatGPT is said to generate on the order of 10 billion tokens of data per day – even before they opened their more compelling GPT-4o model to free users. > Common Crawl (filtered) 410 billion tokens 60% of GPT-3 training data but only 44% of it was used i.e. 0.44 epochs (from the paper published May 28, 2020) From Aligning language models to follow instructions January 27, 2022 [0] > ...these models can also generate outputs that are untruthful, toxic, or reflect harmful sentiments. This is in part because GPT-3 is trained to predict the next word on a large dataset of Internet…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by simbolit

Sat, 14 Oct 2023 11:37:46 GMT

If I understand correctly, the older method you describe has been replaced by exposing a GPT model to some further training (as opposed to "pre-training") with successful conversations. I think this premiered with the InstructGPT paper: https://arxiv.org/pdf/2203.02155.pdf

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by superkuh

Wed, 19 Jul 2023 21:21:21 GMT

Instruction fine tuning definitely increases the win rate in human judged comparisons but that just means it's better at generating a style humans like. Humans aren't always right. https://arxiv.org/pdf/2203.02155.pdf Page 56/68 in table 14 it looks like the things the fine tunes beat the base models at are basically HellaSwag and the ones that use human evaluations. Otherwise base gpt models are winning. And just before that they're discussing how the fine-tunes do cause performance regressions on page 55.

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by senko

Wed, 21 Jun 2023 15:11:47 GMT

Blog post from Huggingface: https://huggingface.co/blog/rlhf Webinar on the same topic (from same HF folks): https://www.youtube.com/watch?v=2MBJOuVq380&t=496s RLHF as used by OpenAI in InstructGPT (predecessor to ChatGPT): https://arxiv.org/abs/2203.02155 (academic paper, so much denser than the above two resources)

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by hansvm

Wed, 21 Jun 2023 15:11:14 GMT

It's not the first paper on the topic IIRC, but OpenAI's InstructGPT paper [0] is decent and references enough other material to get started. The key idea is that they're able to start with large amounts of relatively garbage unsupervised data (the internet), and use that model to cheaply generate decent amounts of better data (ranking generated content rather than spending the man-hours to actually write good content). The other details aren't too important. [0] https://arxiv.org/abs/2203.02155

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by vellum

Wed, 31 May 2023 01:38:27 GMT

From OpenAI's RLHF paper[1]: "By default, when we train a PPO model on our API distribution, it suffers from an “alignment tax”, as its performance on several public NLP datasets decreases." On the HELM[2] site, you can see accuracy benchmarks for InstructGPT <OpenAI model> vs baseline models. The InstructGPT models perform worse on a lot of benchmarks. 1 - https://arxiv.org/pdf/2203.02155.pdf 2 - https://crfm.stanford.edu/helm/v0.1.0/?group=question_answer...

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by joshka

Sun, 28 May 2023 00:22:00 GMT

There's a good slide I saw in Andrej Karpathy's talk[1] at build the other day. It's from a paper talking about training for InstructGPT[2]. Direct link to the figure[3]. The main instruction for people doing the task is: "You will also be given several text outputs, intended to help the user with their task. Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless. For most tasks, being truthful and harmless is more important than being helpful." It had me wondering whether this instruction and the resulting training still had a tendency to train these…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by vellum

Thu, 18 May 2023 06:09:54 GMT

It's the "alignment tax". From OpenAI's RLHF paper[1]: "By default, when we train a PPO model on our API distribution, it suffers from an “alignment tax”, as its performance on several public NLP datasets decreases." On the HELM[2] site, you can see accuracy benchmarks for InstructGPT <OpenAI model> vs baseline models. The InstructGPT models perform worse on a lot of benchmarks. 1 - https://arxiv.org/pdf/2203.02155.pdf 2 - https://crfm.stanford.edu/helm/v0.1.0/?group=question_answer...

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by seydor

Sun, 09 Apr 2023 20:17:03 GMT

I am not an expert but as others have said, the InstructGPT and Alpaca models do that https://arxiv.org/abs/2203.02155# https://crfm.stanford.edu/2023/03/13/alpaca.html

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — HN submission

Sat, 01 Apr 2023 16:59:43 GMT

Training language models to follow instructions with human feedback

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by make3

Wed, 22 Mar 2023 15:42:32 GMT

It just hasn't been prompted or fine-tuned to have the neutral, self effacing personality of ChatGPT. It's doing the pure, "try to guess the most likely next token" task on which they were both trained (https://heartbeat.comet.ml/causal-language-modeling-with-gpt...). ChatGPT is further trained with reinforcement from human feedback to make them more tool-like (https://arxiv.org/abs/2204.05862 & https://openai.com/blog/chatgpt & https://arxiv.org/abs/2203.02155), with a bit of randomness added for variety's sake (https://huggingface.co/blo1g/how-to-generate).

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by jefftk

Tue, 14 Mar 2023 12:51:01 GMT

Reinforcement learning in general didn't come out of AI alignment work, but RHLF in particular did. The initial idea and paper [1] were from AI alignment folks, as was most of the later development [2][3][4][5]. Overview: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoug... [1] Deep Reinforcement Learning from Human Preferences https://arxiv.org/abs/1706.03741 [2] Fine-Tuning Language Models from Human Preferences https://arxiv.org/abs/1909.08593 [3] Learning to summarize from human feedback https://arxiv.org/abs/2009.01325 [4] Recursively Summarizing Books with Human Feedback…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by nl

Fri, 10 Mar 2023 06:27:07 GMT

InstructGPT (which ChatGPT/GPT-3.5) is based on was trained from scratch: > We train three model sizes (1.3B, 6B, and 175B parameters)[1] It does use the same model architecture as GPT-3 though so I guess there is a bit of reasoning there maybe. [1] https://arxiv.org/pdf/2203.02155.pdf

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by nl

Sun, 05 Mar 2023 10:49:24 GMT

Because there are many benchmarks that measure different things. You need to look at the benchmark that reflects your specific interest. So in this case ("I wasn't impressed that 30B didn't seem to know who Captain Picard was") the closest relevant benchmark they performed is MMLU (Massive Multitask Language Understanding"[1]. In the LLAMA paper they publish a figure of 63.4% for the 5-shot average setting without fine tuning on the 65B model, and 68.9% after fine tuning. This is significantly better that the original GPT-3 (43.9% under the same conditions) but as they note: > "[it is] still…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by est

Wed, 15 Feb 2023 10:10:30 GMT

I kind of doubt it? https://arxiv.org/abs/2203.02155 it says > outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by yayr

Sat, 11 Feb 2023 23:43:32 GMT

it is a result of trying to align the AI model after it has been trained using self-supervised learning with an addition human feedback step. OpenAI has a model / process called InstructLM or InstructGPT see e.g. https://arxiv.org/pdf/2212.10496.pdf https://arxiv.org/pdf/2203.02155.pdf also an approach to solve the alignment problem. https://en.wikipedia.org/wiki/AI_alignment

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by mattalex

Sat, 04 Feb 2023 21:43:09 GMT

Instruction tuning mostly relies on the quality of the data you put into the model. This makes it different from traditional language model training: essentially you take one of these existing hugely expensive models (there are lots of them already out there), and tune them specifically on high quality data. This can be done on a comparatively small scale, since you don't need to train trillions of words, but only train on the smaller high quality data (even openai didn't have a lot of that). In fact, if you look at the original paper https://arxiv.org/pdf/2203.02155.pdf Figure 1, you can see…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by binarymax

Sat, 04 Feb 2023 12:06:01 GMT

Read the InstructGPT paper, it’s quite good: https://arxiv.org/abs/2203.02155

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — HN submission

Wed, 18 Jan 2023 21:41:52 GMT

Training language models to follow instructions with human feedback [pdf]

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by nl

Wed, 04 Jan 2023 22:34:06 GMT

> Technically, they aren't trained that way, their opposite recognition AI is trained that way. No, they are trained that way. These aren't GANs or anything resembling that. There is a reinforcement model that is build from expert human guidance which is then used to fine-tune the LM. From the HuggingFace explainer page I linked above: [there are] three core steps: 1. Pretraining a language model (LM), 2. gathering data and training a reward model, and 3. fine-tuning the LM with reinforcement learning. Read https://arxiv.org/abs/2203.02155 if you want all the details (although the HF…

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by imranq

Fri, 23 Dec 2022 23:32:17 GMT

chatGPT is simply a GPT model fine-tuned in an instruction oriented way to align with human-rated helpfulness, truthfulness, and harmlessness as a reward signal. The diagram on the chatGPT blogpost is literally the same diagram in the March 2022 paper: https://arxiv.org/pdf/2203.02155.pdf So it seems like Google could easily replicate this technology, but its difficult to deploy in the real world without brand damage.

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by sinenomine

Mon, 12 Dec 2022 19:21:06 GMT

Nope! I meant specifically https://arxiv.org/abs/2203.02155

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by blackbear_

Fri, 09 Dec 2022 07:18:26 GMT

> they might have a big base of predefined response "templates" This could the result of overfitting on the human feedback that it was trained on [1]. [1] https://arxiv.org/pdf/2203.02155.pdf

Paper and discussion history. A mention is not an endorsement.

Training language models to follow instructions with human feedback — HN submission

Mon, 05 Dec 2022 20:45:51 GMT

Training language models to follow instructions with human feedback

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training language models to follow instructions with human feedback — mentioned by espadrine

Wed, 26 Oct 2022 12:08:12 GMT

BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh. The dark secret of GPT-3 is that…

Paper and discussion history. A mention is not an endorsement.