Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback · HackerPapers discussions

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — mentioned by BoorishBears

Tue, 12 Sep 2023 05:22:11 GMT

If you ask an LLM to complete a sentence like '[Insert name] stole the fruit (true/false):<answer>' An aligned LLM will be biased towards refusing to answer at all with something like: "I can't tell you because I don't know them." An "uncensored" LLM will very happily return <"true"> or <"false"> with a probability attached to each. Even OpenAI's GPT-3 does with a low enough temperature. _ Of course, LLM attention doesn't work like that. The tokens are just a bag of numbers: - The fact the name 'John' is mentioned in the Bible a lot affects the distribution when you ask if any John stole…

Paper and discussion history. A mention is not an endorsement.

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — mentioned by keskival

Wed, 30 Aug 2023 17:12:53 GMT

It is known that alignment tax affects smaller models but in larger models (>~100B parameters) the "tax" starts to become negative at least when trained in RLHF: https://arxiv.org/pdf/2204.05862.pdf The largest Llama2 model has 70B parameters. They ran the experiment with 7B Llama2.

Paper and discussion history. A mention is not an endorsement.

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — HN submission

Fri, 04 Aug 2023 04:02:21 GMT

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — mentioned by make3

Wed, 22 Mar 2023 15:42:32 GMT

It just hasn't been prompted or fine-tuned to have the neutral, self effacing personality of ChatGPT. It's doing the pure, "try to guess the most likely next token" task on which they were both trained (https://heartbeat.comet.ml/causal-language-modeling-with-gpt...). ChatGPT is further trained with reinforcement from human feedback to make them more tool-like (https://arxiv.org/abs/2204.05862 & https://openai.com/blog/chatgpt & https://arxiv.org/abs/2203.02155), with a bit of randomness added for variety's sake (https://huggingface.co/blo1g/how-to-generate).

Paper and discussion history. A mention is not an endorsement.