HackerPapers

arXiv:2210.07128cs.CL

Language Models of Code are Few-Shot Commonsense Learners

Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, Graham Neubig

Abstract

We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event -- or a reasoning-graph. To employ large language models (LMs) for this task, existing approaches ``serialize'' the output graph as a flat list of nodes and edges. Although feasible, these serialized graphs strongly deviate from the natural language corpora that LMs were pre-trained on, hindering LMs from generating them correctly. In this paper, we show that when we instead frame structured commonsense reasoning tasks as code generation tasks, pre-trained LMs of code are better structured commonsense reasoners than LMs of natural language, even when the downstream task does not involve source code at all. We demonstrate our approach across three diverse structured commonsense reasoning tasks. In all these natural language tasks, we show that using our approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task (e.g., T5) and other strong LMs such as GPT-3 in the few-shot setting.

arXiv comments
EMNLP 2022
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 6 comments across 6 threads, from May 2023 to Sep 2023. Found in 3 HN submissions from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Can programming languages boost each other via instruction tuning? 1 comment

    • It was already known that training on code improves performance on all reasoning tasks[1], so this is not really surprising. [1] https://arxiv.org/abs/2210.07128

      wcoenenView on HNParent context

  2. AI Safety and the Age of Dislightenment 1 comment

    • 1. LLMs are general pattern machines. They can generalize and complete in context to a wide variety of complex non linguistic patterns.https://general-pattern-machines.github.io/ 2. LLMs see positive transfer in multi lingual capability. For example, an LLM trained on 500B…

      famouswafflesView on HNParent context

  3. Understanding GPT tokenizers 1 comment

    • >I suspect we're going to discover at some point, or maybe OpenAI already did, that training on code isn't just a neat trick to get an LLM that can knock out scripts. This is a thing that's already fairly well known https://arxiv.org/abs/2210.07128

      famouswafflesView on HNParent context

  4. Ask HN: Why can GPT4 en/decode base64? 1 comment

    • GPT-4 doesn't really fail at simple arithmetic. But it's a harder task to learn because arithmetic doesn't encode information about its solution in preceding context. "9383 + 3545" or "is any of the following numbers a prime, 96885, 66576, 4766 ?" doesn't actually tell you…

      famouswafflesView on HNParent context

  5. Language models still struggle with the concept of negation 1 comment

    • LLMs trained on code reason better. Perform better on reasoning benchmarks even if the benchmarks have nothing to do with code. You're wrong. https://arxiv.org/abs/2210.07128

      famouswafflesView on HNParent context

  6. LLMs are not greater than the sum of their parts: researchers 1 comment

    • Just speculating, but this looks like precisely the sort of result one might expect from fine-tuning models on chain-of-thought-prompted interactions like those described by Wei et al. in 2022[1]. Alternatively, as Madaan et al. show in their 2022 paper[2], this may simply be…

      cosmojgView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.