HackerPapers

arXiv:2412.08905cs.CL

Phi-4 Technical Report

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, Yi Zhang

Abstract

We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 4 threads, from Dec 2024 to Jan 2025. Found in 3 HN submissions from Dec 2024 to Dec 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Phi 4 available on Ollama 1 comment

    • Phi-4's architecture changed slightly from Phi-3.5 (it no longer uses a sliding window of 2,048 tokens [1]), causing a change in the hyperparameters (and ultimately an error at inference time for some published GGUF files on Hugging Face, since the same architecture…

      jmorganView on HNParent context

  2. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023) 1 comment

    • For some interesting context: this paper was a precursor to all the work on synthetic data at Microsoft Research that lead to the Phi series of SLMs. [1] It was an important demonstration of what carefully curated and clean data could do for language models. 1…

      osaarikiView on HNParent context

  3. Ask HN: Why is Ilya saying data is limited when the whole world is data? 1 comment

    • There's a ton of recent work on data curation / synthetic data generation that shows that smaller high quality datasets go a lot further than scaling up on noisy web data. The scaling law plots are log scale so to get more juice with naive scaling we'd need to invest…

      m_keView on HNParent context

  4. Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning 3 comments

    • It's easy to argue that Llama-3.3 8B performs better than GPT-3.5. Compare their benchmarks, and try the two side-by-side. Phi-4 is yet another step towards a small, open, GPT-4 level model. I think we're getting quite close. Check the benchmarks comparing to GPT-4o on the first…

      lappaView on HNParent context

    • The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on…

      simonwView on HNParent context

    • Model releases without comprehensive coverage of benchmarks make me deeply skeptical. The worst was the gpt4o update in November. Basically a 2 liner on what it is better at and in reality it regressed in multiple benchmarks. Here we just get MMLU, which is widely known to be…

      zurferView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.