Orca: Progressive Learning from Complex Explanation Traces of GPT-4 · HackerPapers discussions

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — mentioned by throwuwu

Tue, 04 Jul 2023 15:43:51 GMT

My understanding is that Microsoft research has already published a paper where they used synthetic chat interactions of the same form that chatGPT uses to train a new model. GPT4 could be used to select the best interactions from which to create a training set. I’d be very, very surprised if OpenAI hasn’t already been doing this internally. https://arxiv.org/pdf/2306.02707.pdf

Paper and discussion history. A mention is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Sat, 01 Jul 2023 16:08:09 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — mentioned by throwaway4aday

Fri, 30 Jun 2023 19:28:04 GMT

That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem. Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the…

Paper and discussion history. A mention is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — mentioned by machdiamonds

Wed, 21 Jun 2023 08:20:51 GMT

From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced not only by the type of questions users ask but also by their preference for succinctness in responses. When we venture into the realm of more complex tasks, such as code generation, GPT4 unequivocally outshines all open source models. That said, the advancements in models like Orca and the "Textbooks are all you need"…

Paper and discussion history. A mention is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — mentioned by emadm

Sun, 18 Jun 2023 18:29:08 GMT

The ability to run it on your own private data and train/distill it from the larger models is very useful (see Orca for example: https://arxiv.org/abs/2306.02707). Most of the valuable data in the world is private data unsuitable to send to GPT4 and other proprietary models.

Paper and discussion history. A mention is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Mon, 12 Jun 2023 19:04:59 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

113 points; 29 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Sat, 10 Jun 2023 13:11:41 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Fri, 09 Jun 2023 18:46:21 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — mentioned by kytazo

Fri, 09 Jun 2023 10:38:32 GMT

Genuine link: https://arxiv.org/pdf/2306.02707.pdf HN apparently won't let duplicate posts and since the first submission didn't really gain any traction thought I'd resubmit a video version of it so people catch up with what is most likely the state of the art in open source. As stated in the papper MS is going to release the model in a similar manner to llama, so this is definitely exciting news for many.

Paper and discussion history. A mention is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Wed, 07 Jun 2023 16:50:47 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

11 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Tue, 06 Jun 2023 17:59:23 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

4 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 — HN submission

Tue, 06 Jun 2023 08:08:30 GMT

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.