HackerPapers

arXiv:2210.11610cs.CL

Large Language Models Can Self-Improve

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han

Abstract

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 9 identifiable commenters in 9 comments across 8 threads, from Jan 2023 to May 2023. Found in 3 HN submissions from Nov 2022 to Feb 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Constitutional AI: RLHF on Steroids 1 comment

    • > This result feels like creepy perpetual motion. It’s like they’re teaching the AI ethics by making it write an ethics textbook and then read the textbook it just wrote. Is this a free lunch? Shouldn’t it be impossible for the AI to teach itself any more ethics than it started…

      taneqView on HNParent context

  2. GPT-3 will ignore tools when it disagrees with them 1 comment

    • I don't think it can be so easily dismissed. AlphaZero contributed lots, for example reproducing opening theory from a blank slate. For chess that's significant. But we don't care about chess for this discussion about GPT. Self-learning and self-play would be interesting because…

      kzrdudeView on HNParent context

  3. Poisoning web-scale training datasets is practical 1 comment
  4. Eventually ChatGPT will eat itself 2 comments

    • Here’s a more directly comparable example: https://arxiv.org/abs/2210.11610

      moyixView on HNParent context

    • I have seen no evidence for that, only the opposite: https://arxiv.org/abs/2210.11610 Intuitively, training a simple enough linear statistical model with its own output should be a NOP. But LLMs are anything but simple models, so I think the non-linearities may be synthesizing…

      jbenjosephView on HNParent context

  5. Are we racing toward AI catastrophe? 1 comment

    • There is evidence to suggest that LLMs have recently passed a threshold where their performance can be improved by training on their own output. https://arxiv.org/abs/2210.11610 I expect your concerns are likely unwarranted.

      valineView on HNParent context

  6. ChatGPT is a blurry JPEG of the web 1 comment

    • >But, it's not implausible that some future model will simply have a superhuman learning rate and a superhuman ability to distinguish "right" from "wrong" - this paragraph will look downright prophetic then. There is already a paper for that: https://arxiv.org/abs/2210.11610…

      anabisView on HNParent context

  7. John Carmack’s ‘Different Path’ to Artificial General Intelligence 1 comment

    • There are ways that LLM's can self-improve, such as in this paper: https://arxiv.org/abs/2210.11610 I would speculate that there are more ways to train on logical consistency of the output, and improve the models further.

      vacuumclView on HNParent context

  8. ChatGPT Can't Kill Anything Worth Preserving 1 comment

    • It was not a language model, but AlphaZero played against itself millions of games and trained on the data it generated, without seeing any human games. And it reached super human level, of course. Why was this possible? The game itself acted as an anti-bullshit filter. So you…

      visargaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.