Crosslingual Generalization through Multitask Finetuning · HackerPapers discussions

Crosslingual Generalization through Multitask Finetuning — mentioned by vvrm

Fri, 20 Jan 2023 19:17:10 GMT

Thanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following: > rough steps: > 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ? > 2. get on a call with the sales teams of major cloud providers to procure a…

Paper and discussion history. A mention is not an endorsement.

Crosslingual Generalization through Multitask Finetuning — mentioned by karpathy

Wed, 18 Jan 2023 00:21:23 GMT

rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference…

Paper and discussion history. A mention is not an endorsement.