The Llama 3 Herd of Models · HackerPapers discussions

The Llama 3 Herd of Models — mentioned by sva_

Sun, 27 Sep 2026 00:33:19 GMT

Compare to https://arxiv.org/abs/2407.21783

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by JacobAsmuth

Mon, 27 Jul 2026 20:06:20 GMT

1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by robrenaud

Wed, 14 May 2025 15:33:11 GMT

Meta synthetically generated lots of PHP from Python for Llama 3 for training purposes. Meta writes a crazy amount of PHP internally. Translation tends to be way easier than unconstrained generation for LLMs. But if you can translate and filter a large amount of code, you can learn to generate. If you also translate and run the unittests, you get another layer of error checking. https://arxiv.org/abs/2407.21783 See figure 8.

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by rafram

Thu, 06 Mar 2025 16:11:56 GMT

Right, the original LLaMA paper. LLaMA 2 and 3 are significantly more capable models trained on orders of magnitude more data, and those papers notably do not say where the data comes from. The LLaMA 3 paper helpfully mentions that "Much of the data we utilize is obtained from the web," so I guess that's better than nothing! LLaMA 2: https://arxiv.org/pdf/2307.09288 LLaMA 3: https://arxiv.org/pdf/2407.21783

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by aubanel

Thu, 20 Feb 2025 13:19:40 GMT

> Which shows that "massive scaling", even enormous, gigantic scaling, doesn't improve intelligence one bit; it improves scope, maybe, or flexibility, or coverage, or something, but not "intelligence". Do you have any data to support 1. That grok is not more intelligent than previous models (you gave one anecdotal datapoint), and 2. That it was trained on more data than other models like o1 and Claude-3.5 Sonnet? All datapoints I have support the opposite: scaling actually increases intelligence of models. (agreed, calling this "intelligence" might be a stretch, but alternative definitions…

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by dongecko

Tue, 11 Feb 2025 15:56:13 GMT

Yes, they can. Meta described a clever way in their paper on training Llama3 [1] (in section about factuality). The idea is to sample several answers to a question that you know the answer to. Let an LLM decide if the given answers are different from your known truth. If so, you found a question, that you can train your LLM in the next post-training round to answer with "I don't know". Do that a couple hundred of times and your LLM will identify neurons that indicate doubt and from here on have the ability to answer with "I don't know". [Edit] The article here also mentions a paper [2] that…

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — HN submission

Mon, 09 Dec 2024 06:56:28 GMT

The Llama 3 Herd of Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Llama 3 Herd of Models — mentioned by ryao

Wed, 04 Dec 2024 04:07:03 GMT

Facebook did a technical paper where they described their training cluster and the sheer amount of complexity is staggering. That said, the original poster was interested in inferencing, not training. https://arxiv.org/pdf/2407.21783

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by curious_cat_163

Wed, 02 Oct 2024 14:48:13 GMT

Thanks for sharing your thoughts. I have a somewhat more optimistic take: We will create some form of artificial intelligence (AI) in the next decade. However, the recent (i.e. over last five years) advances in natural language processing are not the ones that automatically lead to that. I think we need more fundamental breakthroughs than attention-based neural networks and all the tricks that have gone into say a Llama3. [1] Even if we did develop an AI more aggressively, say by 2030, I am not sure if that automatically leads to extinction. Here, my optimism is guided by the last few times…

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by johntb86

Tue, 20 Aug 2024 18:30:12 GMT

GPT-4o has seen a lot of examples of what people write about coding on the web, what code exists, and what tasks people want to do with that. But that general data doesn't include full process of coding - get a bug report, look at a codebase beforehand, make changes to the codebase, test them and see exactly what bugs they have, and iterate. That process is what SWE-bench tests. It's possible OpenAI did some coding fine-tuning themselves; Meta's Llama 3 paper [0], section 4.3.1 mentions what sort of work is needed. However, anything OpenAI did is based on their own tooling and set of…

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — mentioned by tkgally

Sat, 03 Aug 2024 02:22:59 GMT

I haven’t tried Llama 3.1 405B much yet, but I was impressed by the detail and openness of Meta’s paper about it [1]. This interview with Meta’s head of product management for generative AI adds some interesting perspective. [1] https://arxiv.org/abs/2407.21783

Paper and discussion history. A mention is not an endorsement.

The Llama 3 Herd of Models — HN submission

Thu, 01 Aug 2024 14:21:40 GMT

The Llama 3 Herd of Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.