Language Models of Code are Few-Shot Commonsense Learners · HackerPapers discussions

Language Models of Code are Few-Shot Commonsense Learners — mentioned by wcoenen

Fri, 01 Sep 2023 15:59:05 GMT

It was already known that training on code improves performance on all reasoning tasks[1], so this is not really surprising. [1] https://arxiv.org/abs/2210.07128

Paper and discussion history. A mention is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — mentioned by famouswaffles

Tue, 11 Jul 2023 15:45:22 GMT

1. LLMs are general pattern machines. They can generalize and complete in context to a wide variety of complex non linguistic patterns.https://general-pattern-machines.github.io/ 2. LLMs see positive transfer in multi lingual capability. For example, an LLM trained on 500B tokens of English and 10B tokens of French will not speak in french like a model trained on only 10B tokens of French. What will happen is that the model will be nearly as competent in French as it is in English https://arxiv.org/abs/2108.13349 3. Language models of code reason better even if the benchmarks have nothing to…

Paper and discussion history. A mention is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — mentioned by famouswaffles

Fri, 09 Jun 2023 00:01:19 GMT

>I suspect we're going to discover at some point, or maybe OpenAI already did, that training on code isn't just a neat trick to get an LLM that can knock out scripts. This is a thing that's already fairly well known https://arxiv.org/abs/2210.07128

Paper and discussion history. A mention is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — HN submission

Mon, 15 May 2023 20:03:36 GMT

Language Models of Code Are Few-Shot Commonsense Learners

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — HN submission

Sat, 13 May 2023 22:54:26 GMT

Language Models of Code Are Few-Shot Commonsense Learners

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — mentioned by famouswaffles

Sat, 13 May 2023 14:02:48 GMT

GPT-4 doesn't really fail at simple arithmetic. But it's a harder task to learn because arithmetic doesn't encode information about its solution in preceding context. "9383 + 3545" or "is any of the following numbers a prime, 96885, 66576, 4766 ?" doesn't actually tell you anything that would inform the answer. You go to school and you learn the required set of steps for solving these problems. On the hand, for "John is smiling so he is _____", the preceding context screams happy as a very likely choice. Preceding context actually helps finding the solution rather than being the equivalent of…

Paper and discussion history. A mention is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — HN submission

Sat, 13 May 2023 13:00:53 GMT

Code trained LLMs reason better, on benchmarks that have nothing to do with code

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — mentioned by famouswaffles

Sat, 13 May 2023 12:47:00 GMT

LLMs trained on code reason better. Perform better on reasoning benchmarks even if the benchmarks have nothing to do with code. You're wrong. https://arxiv.org/abs/2210.07128

Paper and discussion history. A mention is not an endorsement.

Language Models of Code are Few-Shot Commonsense Learners — mentioned by cosmojg

Tue, 09 May 2023 00:55:35 GMT

Just speculating, but this looks like precisely the sort of result one might expect from fine-tuning models on chain-of-thought-prompted interactions like those described by Wei et al. in 2022[1]. Alternatively, as Madaan et al. show in their 2022 paper[2], this may simply be the result of larger language models having seen more code, and thus structured reasoning, in their training data. [1] https://arxiv.org/abs/2201.11903 [2] https://arxiv.org/abs/2210.07128

Paper and discussion history. A mention is not an endorsement.