Textbooks Are All You Need II: phi-1.5 technical report · HackerPapers discussions

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by visarga

Thu, 11 Jan 2024 21:31:48 GMT

Yes. > We follow the “Textbooks Are All You Need” approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. https://arxiv.org/pdf/2309.05463.pdf

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by dmezzetti

Sun, 07 Jan 2024 00:29:58 GMT

Couple other interesting papers from 2023 to add, specifically for small language models. TinyStories: https://arxiv.org/abs/2305.07759 Phi-1.5: https://arxiv.org/abs/2309.05463 Both these papers proved that with good/concise data we can get better performance with smaller parameter counts.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by behohippy

Sat, 06 Jan 2024 12:16:04 GMT

It's probably an evolution of the phi-1/1.5 "Textbooks are all you Need" training method: https://arxiv.org/abs/2309.05463

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by yyyk

Mon, 27 Nov 2023 09:12:52 GMT

Maybe this will help? https://huggingface.co/datasets/roneneldan/TinyStories Also note https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by Philpax

Fri, 24 Nov 2023 17:08:10 GMT

The idea is that you can standardise the quality of the training data by taking source articles and synthesizing new data with the same "voice" and structure, as well as being able to collate insights from multiple sources. This is the line of thinking behind the Phi lineup of models [0], as well as efforts to generate synthetic textbooks for training [1]. [0]: https://arxiv.org/abs/2309.05463 [1]: https://twitter.com/ocolegro/status/1712327588255809667

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by Philpax

Tue, 14 Nov 2023 01:51:06 GMT

> AI cannot sustain itself trained on AI work. This isn’t true. You can train LLMs entirely on synthetic data and get strong results. [0] > If new languages, engines etc pop up it cannot synthesize new forms of coding without that code having existed in the first place. You can describe the semantics to a LLM, have it generate code, tell it what went wrong (i.e. with compiler feedback), and then train on that. For an example of this workflow in a different context, see [1]. > And most importantly, it cannot fundamentally rationalize about what code does or how it functions. Most competent…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by visarga

Tue, 10 Oct 2023 20:17:25 GMT

> if the training data does not contain a set of generalizable text applicable to a particular domain, a language model is unlikely to make a correct inference when confronted with a novel version of a similar situation. True. But look at the Phi-1.5 model - it punches 5x above its weight limit. The trick is in the dataset: > Our training data for phi-1.5 is a combination of phi-1’s training data (7B tokens) and newly created synthetic, “textbook-like” data (roughly 20B tokens) for the purpose of teaching common sense reasoning and general knowledge of the world (science, daily activities…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by isaacfung

Fri, 22 Sep 2023 11:37:17 GMT

LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463 Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models https://arxiv.org/abs/2307.14430

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — mentioned by isaacfung

Mon, 18 Sep 2023 05:25:16 GMT

Quality matters just as much as quantity. LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — HN submission

Tue, 12 Sep 2023 16:18:51 GMT

Textbooks Are All You Need II: phi-1.5 technical report

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Textbooks Are All You Need II: phi-1.5 technical report — HN submission

Tue, 12 Sep 2023 14:55:46 GMT

The tide is shifting: 1.3B outperforms 7B Llama 2

63 points; 15 discussion replies on HN. Paper and source history. A submission is not an endorsement.