Scaling Instruction-Finetuned Language Models · HackerPapers discussions

Scaling Instruction-Finetuned Language Models — mentioned by famouswaffles

Tue, 09 May 2023 00:45:43 GMT

>OpenAI hasn’t told us why ChatGPT is so much better than other models Here's the thing.... It's not. Unless you restrict the pool to models you know about. https://crfm.stanford.edu/helm/latest/?group=core_scenarios And Again, chatGPT was not the general SOTA LLM. https://arxiv.org/abs/2210.11416 So in the research world with detailed papers explaining what and what not, we had models that were better.

Paper and discussion history. A mention is not an endorsement.

Scaling Instruction-Finetuned Language Models — mentioned by famouswaffles

Mon, 27 Feb 2023 16:09:54 GMT

GPT 3.5 is definitely not comparable to Palm. You can see some benchmarks here. https://crfm.stanford.edu/helm/latest/?group=core_scenarios For example, text-davinci-3 is 56.9 on MMLU, Flan-Palm is 75. https://arxiv.org/abs/2210.11416

Paper and discussion history. A mention is not an endorsement.

Scaling Instruction-Finetuned Language Models — mentioned by lossolo

Thu, 08 Dec 2022 22:46:16 GMT

11B parameters, so I guess it's FLAN-T5[1]? combined with retrieval transformer (something like RETRO[2])? 1. https://arxiv.org/abs/2210.11416 2. https://arxiv.org/abs/2112.04426

Paper and discussion history. A mention is not an endorsement.

Scaling Instruction-Finetuned Language Models — HN submission

Mon, 05 Dec 2022 20:41:59 GMT

Scaling Instruction-Finetuned Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Scaling Instruction-Finetuned Language Models — mentioned by espadrine

Wed, 26 Oct 2022 12:08:12 GMT

BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh. The dark secret of GPT-3 is that…

Paper and discussion history. A mention is not an endorsement.