Orca 2: Teaching Small Language Models How to Reason · HackerPapers discussions

Orca 2: Teaching Small Language Models How to Reason — mentioned by alecco

Tue, 21 Nov 2023 13:41:40 GMT

> Progressive Learning: We start with LLaMA-2-7B or LLaMA-2-13B checkpoint and finetune it on the train split of FLAN-v2 dataset for one epoch. Note that FLAN-v2 dataset contains both zero-shot and few-shot problems. We then train on 5 million ChatGPT data from Orca 1 for 3 epochs. Then we train on the combination of 1 million GPT-4 data from Orca 1 and Orca 2’s 817K data for 4 epochs. I think people are missing why they are comparing against Llama-2 13B/70B. They improved Llama-2 7B/13B and reach the level of a 5-10x larger model of the same base. This is huge. Models on HF.…

Paper and discussion history. A mention is not an endorsement.

Orca 2: Teaching Small Language Models How to Reason — HN submission

Tue, 21 Nov 2023 10:16:35 GMT

Orca 2: Teaching Small Language Models How to Reason

310 points; 80 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Orca 2: Teaching Small Language Models How to Reason — HN submission

Tue, 21 Nov 2023 05:38:58 GMT

Orca 2: Teaching Small Language Models How to Reason

18 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.