OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization · HackerPapers discussions

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — mentioned by vvrm

Fri, 20 Jan 2023 19:17:10 GMT

Thanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following: > rough steps: > 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ? > 2. get on a call with the sales teams of major cloud providers to procure a…

Paper and discussion history. A mention is not an endorsement.

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — mentioned by karpathy

Wed, 18 Jan 2023 00:21:23 GMT

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference…

Paper and discussion history. A mention is not an endorsement.

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — mentioned by espadrine

Tue, 10 Jan 2023 16:55:31 GMT

In terms of quality, I think BLOOMZ, or mT0, are the best open-source ones. The non-finetuned BLOOM does not appear favorably (in English) compared to GLM or OPT, which both have published weights: https://crfm.stanford.edu/helm/v0.1.0/?group=mmlu and Flan-T5 is above OPT-IML: https://arxiv.org/pdf/2212.12017.pdf > Is the future going to be controlled by big corporations who own the models themselves? On this subject, there is an effort stemming from BigScience to build an open, distributed inference network, so that people that don’t have enough GPUs at home can contribute theirs and get…

Paper and discussion history. A mention is not an endorsement.