Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models · HackerPapers discussions

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by visarga

Sat, 06 Apr 2024 05:39:08 GMT

I don't think this paper goes deep enough, they don't consider for example structured diversity seeding for LLMs, which entails prompting the model to combine diverse entities, knowledge and skills selected at random from a list. Random combinations of conditioning context can lead the model to hyper-diversity. The Skill-Mix paper shows how this can work. https://arxiv.org/abs/2310.17567 LLMs have been trained on everything, when they are also prompted by diverse people with their own tasks and styles it won't lead to knowledge collapse. Two diverse worlds meeting each other in chat LLM…

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by visarga

Fri, 15 Mar 2024 14:51:50 GMT

Maybe the Skill Mix paper is relevant here. They define a list of 100 skills, and then randomly sample tuples of n skills (usually less than 6) and generate a test example using those skills. Apparently only GPT-4 (at the time of the paper) was able to compose 5 skills, the other models just 3 or 2. Beyond 5 skills even GPT-4 was doing much worse. The interesting finding of the paper is that GPT-4 couldn't have seen all the (topic, skill-tuple) combinations in the training set. If you have 10,000 examples on a topic, and use 5 out of 100 skills, you would need 100^5 training examples to cover…

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by sp332

Thu, 08 Feb 2024 19:45:10 GMT

Well, sometimes. Sometimes not. https://arxiv.org/abs/2310.17567

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — HN submission

Tue, 06 Feb 2024 03:20:39 GMT

Skill-Mix: A Flexible and Expandable Family of Evaluations for AI Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by visarga

Thu, 01 Feb 2024 06:37:04 GMT

> Would you say that GPT-4 can reason now? Let's assume reasoning entails going beyond the stochastic parrot level. Can LLMs have skills not demonstrated in the training set? Here is a paper demonstrating that GPT-4 can combine up to 5 skills from a set of 100, effectively covering 100^5 tuples of skills, while only seeing much fewer combinations in training on a specific topic. > simple probability calculations indicate that GPT-4's reasonable performance on k=5 is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not…

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by visarga

Thu, 01 Feb 2024 05:15:02 GMT

It's not perfect but it can reason, or solve many more situations than trained on. Here is a paper demonstrating that GPT-4 can combine up to 5 skills from a set of 100, effectively covering 100^5 tuples of skills, while only seeing much fewer combinations in training on a specific topic. > simple probability calculations indicate that GPT-4's reasonable performance on k=5 is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training https://arxiv.org/abs/2310.17567

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by visarga

Tue, 23 Jan 2024 17:25:06 GMT

This article is 3 years old, right before GPT-3's launch. So it doesn't have all the recent info. We're not using meta-learning much, the problem of using few examples to train a model has been solved by few-shot prompting, zero shot, or fine-tuning with a small dataset. The problem with meta-learning is that it tries to back-propagate through the inner loop (the model) to the outer loop (meta-model), and this is hard to do. Since LLMs have such huge and diverse training sets, almost everything is in-distribution. LLMs are conditional probability distributions that can be steered with prompts…

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by svat

Mon, 22 Jan 2024 19:02:57 GMT

What I understood: they assume a hypothetical bipartite graph with text nodes (like hypothetical paragraphs / chunks of texts) on one side, and “skill” nodes on the other. The skills could be things like: • ability to understand the word “because” [causality] • ability to divide two numbers • ability to detect irony Each text node is connected to all the skill nodes that are required to understand it. (This is a hypothetical graph, so we don't need to be able to write down the graph precisely, or define what “understand” or “required” mean here.) Further, each text node is either “successful”…

Paper and discussion history. A mention is not an endorsement.

Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models — mentioned by svat

Mon, 22 Jan 2024 18:56:27 GMT

Quanta is usually careful to link to the actual papers (even if the reader has to be equally careful to find them), so it's surprising that the link is missing here. Another paper (linked from the article) is https://arxiv.org/abs/2310.17567

Paper and discussion history. A mention is not an endorsement.