Phi-4 Technical Report · HackerPapers discussions

Phi-4 Technical Report — mentioned by jmorgan

Sun, 12 Jan 2025 04:16:20 GMT

Phi-4's architecture changed slightly from Phi-3.5 (it no longer uses a sliding window of 2,048 tokens [1]), causing a change in the hyperparameters (and ultimately an error at inference time for some published GGUF files on Hugging Face, since the same architecture name/identifier was re-used between the two models). For the Phi-4 uploaded to Ollama, the hyperparameters were set to avoid the error. The error should stop occurring in the next version of Ollama [2] for imported GGUF files as well In retrospect, a new architecture name should probably have been used entirely, instead of…

Paper and discussion history. A mention is not an endorsement.

Phi-4 Technical Report — mentioned by osaariki

Thu, 02 Jan 2025 20:12:18 GMT

For some interesting context: this paper was a precursor to all the work on synthetic data at Microsoft Research that lead to the Phi series of SLMs. [1] It was an important demonstration of what carefully curated and clean data could do for language models. 1: https://arxiv.org/abs/2412.08905

Paper and discussion history. A mention is not an endorsement.

Phi-4 Technical Report — HN submission

Wed, 25 Dec 2024 12:07:19 GMT

Phi-4 Technical Report

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Phi-4 Technical Report — mentioned by zurfer

Mon, 16 Dec 2024 09:16:16 GMT

Model releases without comprehensive coverage of benchmarks make me deeply skeptical. The worst was the gpt4o update in November. Basically a 2 liner on what it is better at and in reality it regressed in multiple benchmarks. Here we just get MMLU, which is widely known to be saturated and knowing they trained on synthetic data, we have no idea how much "weight" was given to having MMLU like training data. Benchmarks are not perfect, but they give me context to build upon. --- edit: the benchmarks are covered in the paper: https://arxiv.org/pdf/2412.08905

Paper and discussion history. A mention is not an endorsement.

Phi-4 Technical Report — mentioned by simonw

Sun, 15 Dec 2024 23:32:22 GMT

The most interesting thing about this is the way it was trained using synthetic data, which is described in quite a bit of detail in the technical report: https://arxiv.org/abs/2412.08905 Microsoft haven't officially released the weights yet but there are unofficial GGUFs up on Hugging Face already. I tried this one: https://huggingface.co/matteogeniaccio/phi-4/tree/main I got it working with my LLM tool like this: llm install llm-gguf llm gguf download-model https://huggingface.co/matteogeniaccio/phi-4/resolve/main/phi-4-Q4_K_M.gguf llm chat -m gguf/phi-4-Q4_K_M Here are some initial…

Paper and discussion history. A mention is not an endorsement.

Phi-4 Technical Report — mentioned by m_ke

Sun, 15 Dec 2024 18:34:53 GMT

There's a ton of recent work on data curation / synthetic data generation that shows that smaller high quality datasets go a lot further than scaling up on noisy web data. The scaling law plots are log scale so to get more juice with naive scaling we'd need to invest exponentially more resources, and we're at a point where the juice is not worth the squeeze, so people will shift to moving the curve down with new architectures, better curated datasets and test time compute / RL. See: - FineWeb: https://arxiv.org/abs/2406.17557 - Phi-4: https://arxiv.org/abs/2412.08905 - DataComp…

Paper and discussion history. A mention is not an endorsement.

Phi-4 Technical Report — HN submission

Sat, 14 Dec 2024 02:12:02 GMT

Phi-4 Technical Report

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Phi-4 Technical Report — HN submission

Fri, 13 Dec 2024 14:04:47 GMT

Phi-4 Technical Report

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Phi-4 Technical Report — mentioned by lappa

Fri, 13 Dec 2024 05:10:17 GMT

It's easy to argue that Llama-3.3 8B performs better than GPT-3.5. Compare their benchmarks, and try the two side-by-side. Phi-4 is yet another step towards a small, open, GPT-4 level model. I think we're getting quite close. Check the benchmarks comparing to GPT-4o on the first page of their technical report if you haven't already https://arxiv.org/pdf/2412.08905

Paper and discussion history. A mention is not an endorsement.