LLaMA: Open and Efficient Foundation Language Models · HackerPapers discussions

LLaMA: Open and Efficient Foundation Language Models — mentioned by JW_00000

Fri, 07 Feb 2025 14:53:24 GMT

I don't understand why it's even a question that Meta trained their LLM on copyrighted material. They say so in their paper! Quoting from their LLaMMa paper [Touvron et al., 2023]: > We include two book corpora in our training dataset: the Gutenberg Project, [...], and the Books3 section of ThePile (Gao et al., 2020), a publicly available dataset for training large language models. Following that reference: > Books3 is a dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser (Presser, 2020). (Presser, 2020) refers to…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by michaelt

Thu, 13 Jun 2024 09:43:50 GMT

Then, you should check out papers like https://arxiv.org/abs/2302.13971 and https://arxiv.org/abs/2307.09288 In the paper covering the original Llama they explicitly list their data sources in table 1 - including saying that they pretrained on the somewhat controversial books3 dataset. The paper for Llama 2 also explicitly says they don't take data from Meta's products and services; and that they filter out data from sites known to contain a lot of PII. Although it is more coy about precisely what data sources they used, like many such papers are.

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — HN submission

Thu, 28 Mar 2024 20:19:31 GMT

LLaMA: Open and Efficient Foundation Language Models (2023)

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by simonw

Wed, 27 Mar 2024 13:39:19 GMT

The most detailed answer to that I've seen is the original LLaMA paper, which described exactly what that model was trained on (including lots of scraped copyrighted data) https://arxiv.org/abs/2302.13971 Llama 2 was much more opaque about the training data, presumably because they were already being sued at that point (by Sarah Silverman!) over the training data that went into the first Llama! A couple of things I've written about this: - https://simonwillison.net/2023/Aug/27/wordcamp-llms/#how-the... - https://simonwillison.net/2023/Apr/17/redpajama-data/

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by versteegen

Sun, 17 Dec 2023 23:24:32 GMT

Because the training data/model size/compute tradeoff derived from that paper is highly suboptimal (too many parameters) compared to the ones from the later Deepmind scaling laws [1]. And then Meta researchers recommended using even smaller models, to trade-off training- and inference-time compute [2] (which I thought was pretty obvious if you care about more than just benchmarks). [1] https://arxiv.org/abs/2203.15556 Training Compute-Optimal Large Language Models [2] https://arxiv.org/abs/2302.13971 LLaMA: Open and Efficient Foundation Language Models

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by smoldesu

Wed, 27 Sep 2023 22:24:36 GMT

Facebook has been open-sourcing AI research longer than OpenAI has even had the concept of an "investor narrative". I struggle to understand how someone could jump to the conclusion of this being a "scorched earth" maneuver with so many other reasonable explanations. Facebook has a laboratory (FAIR) with a long history of research and releases like this. > If Llama cost hypothetical one billion, they inflict a multiple on that on their competitors with this move while gaining massive technological advantages. If Llama cost a hypothetical one billion, then they amortized the cost over the…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by computerex

Wed, 06 Sep 2023 21:14:48 GMT

Llama: https://arxiv.org/pdf/2302.13971.pdf You can clearly see in table on second page that higher quality data is trained on more than 1 epoch. Most Open LLM's do this.

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by fpgaminer

Wed, 06 Sep 2023 16:53:06 GMT

Llama off the top of my head: https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by v64

Mon, 04 Sep 2023 22:48:24 GMT

The Llama 1 paper [1] was one of the earlier models to question the assumption that more params = better model. Since then they've released Llama 2 and this post is offering more evidence that reinforces their hypothesis. I wouldn't say it was an oversight by other labs that they missed this. It's easier to just increase params on a model over the same training set instead of gathering a larger training set necessary for a smaller model. And at first, increasing model size did seem to be the way forward, but we've since hit diminishing returns. Now that we've hit that point, we've begun…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by salawat

Mon, 10 Jul 2023 14:59:36 GMT

It's in the LLaMa.cpp original research paper. Tis mentioned in the brief. The research paper basically stated it was trained on Bibliotik, and other Internet "shadow library" corpuses. See the reference to the Gao et al, in the linked paper from the article. Paper linked in article: https://arxiv.org/pdf/2302.13971.pdf The LLaMA paper references a paper utilizing a data source compiled by EleutherAI otherwise known as ThePile. URL from the bibliography for that paper points yonder: https://zenodo.org/record/7413426 This act of summarization done in a lovingly amateur nature at no cost to…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by edent

Sun, 09 Jul 2023 21:21:52 GMT

Are we all reading the same complaint? They say: > in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an EleutherAI paper as being put together from “a copy of the contents of the Bibliotik private tracker.” Does that stack up? The Meta Paper - https://arxiv.org/pdf/2302.13971.pdf - says: > We include two book corpora in our training dataset: the Gutenberg Project, which contains books that are in the public domain, and…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by pythux

Sun, 18 Jun 2023 20:46:01 GMT

It is very expensive to train these base models so a smaller size is more practical if you aren’t a big company with hundreds of powerful GPUs at hand. Table 15 from LLaMA paper[1] has some insightful figures: it took 135,168 GPU hours to train the 13B version and a bit more than 1M GPU hours for the 65B version. And we are talking about A100 80GB GPUs here (expensive and scarce). Not everyone can afford these kinds of trainings (especially if it takes a few attempts; e.g. if you’ve got a bug in the tokenizer) [1] https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by londons_explore

Thu, 08 Jun 2023 15:13:36 GMT

I don't think Facebooks moves were heavily strategically thought out. Instead, I think a small team of a few researchers made LLaMA and published a paper about it[1], and then realised to get any citations for their paper they'd need to let other researchers have the weights. Citations are like money in the academic world. Every researcher wants citations. Executives and managers care less about that stuff. [1]: https://arxiv.org/abs/2302.13971

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by ppaattrriicckk

Mon, 22 May 2023 19:15:27 GMT

I love bashing Intel as much as the other person, but looking at the number of tokens LLaMA[1] was trained on (1,400B tokens), the number of parameters of GPT-3[2] vs. the number of tokens it was trained on (175B params vs. 300B), it's not like the announced 1,000B param model is unreasonable. Taking 175/300*1400 yields 816B parameters - which is fairly close to 1,000B. Not outright the most efficient utilization of data, but as others have mentioned, there might still be something left to gain (from not solely optimizing for compute). E.g. look at Figure 9 in [3]. Although the model trained…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by capableweb

Sun, 14 May 2023 13:34:08 GMT

First link: https://github.com/ggerganov/llama.cpp Which in turn has the following as the first link: https://arxiv.org/abs/2302.13971 Is it really quicker to ask here than just browse content for a bit, skimming some text or even using Google for one minute?

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by YetAnotherNick

Wed, 03 May 2023 12:30:10 GMT

They are likely doing some interpolation for 200B or benchmarking it in wrong way. e.g. Hellaswag accuracy for llama 7b is 0.76[1], but it is written 0.56 in the repo. Even at 200B tokens, it is higher than 0.56 for llama looking at the charts. [1]: https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by nl

Mon, 17 Apr 2023 23:28:06 GMT

LLama 65B used 2048 80G A100s for 21 days: > When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM.[1] Note that you probably need to budget for double to triple that because things go wrong and it usually takes multiple starts to get a good training run. Smaller models are cheaper though. [1] https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by sp332

Mon, 17 Apr 2023 21:14:51 GMT

Page 4 https://arxiv.org/abs/2302.13971 When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM. This means that training over our dataset containing 1.4T tokens takes approximately 21 days. At $4/GPU-hour per A100 80GB GPU, that's $4 * 2,048 * 21 * 24 = $4,128,768.

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by eiz

Mon, 17 Apr 2023 21:08:44 GMT

https://arxiv.org/pdf/2302.13971.pdf table 15. 1770394 A100-80GB hours to train the entire model suite at the going rate for cloud 8xA100-80GBs (~$12/hr if you could actually get capacity) is ~$2.6M, under extremely optimistic assumptions. YMMV on bulk pricing ;) "the more you buy the more you save"

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by frozencell

Thu, 13 Apr 2023 19:11:24 GMT

> ChatGPT competitors of EU origin? ChatGPT is not research. If you want real research: https://ai.facebook.com/blog/large-language-model-llama-meta... Edit: end paper is https://arxiv.org/abs/2302.13971 > Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by lhl

Mon, 10 Apr 2023 09:51:09 GMT

It's worth pointing out that size isn't everything. From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals. Those interested in doing more comparisons can look at https://crfm.stanford.edu/helm/latest/?group=core_scenarios and https://paperswithcode.com/paper/llama-open-and-efficient-fo... to see where it sits (with some GPT 3.5 and 4 numbers here: https://paperswithcode.com/paper/gpt-4-technical-report-1) I'd agree the secret sauce for how great the newest services perform is probably in the…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by yreg

Tue, 04 Apr 2023 08:31:56 GMT

Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0]. Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right? [0] https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by sp332

Sat, 01 Apr 2023 00:42:55 GMT

Check out the graph on page 3 of this PDF: https://arxiv.org/abs/2302.13971 The 33B model started beating the 7B when it had been trained on only 1/3 as much data. And then they kept training it to 40% more than the total that 7B was trained on. It's better.

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by dchest

Fri, 31 Mar 2023 22:50:25 GMT

https://arxiv.org/abs/2302.13971

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by nl

Thu, 30 Mar 2023 22:40:40 GMT

I posted the wrong link sorry! https://arxiv.org/pdf/2302.13971.pdf (the Llama paper), table 7.

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by imranq

Thu, 30 Mar 2023 10:56:27 GMT

The stats shown are pretty compelling that this transformer + RNN approach works. Looking back at the Llama paper (https://arxiv.org/pdf/2302.13971.pdf), many of the results are comparable between llama and RWKV. E.g. Llama 13bn scores 80.1 on PIQA and RWKV scores 77.5 I wonder if RLHF would boost performance for this architecture in the same way

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by JW_00000

Wed, 29 Mar 2023 21:36:33 GMT

The LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included. Also almost all English books must be included. Newspaper archives is about the only thing I see as missing, as well as more non-English sources. [1] https://arxiv.org/abs/2302.13971

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by corford

Wed, 29 Mar 2023 10:20:14 GMT

And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by gpm

Tue, 28 Mar 2023 17:40:32 GMT

Is there a benchmark comparing the two that I missed? Edit: The huggingface page has 0-shot benchmarks which you can compare against the llama paper https://huggingface.co/cerebras/Cerebras-GPT-13B https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by rnosov

Fri, 24 Mar 2023 09:01:31 GMT

There are benchmarks in the original LLaMA paper[1]. Specifically, on page 4 LLaMA 13B seems to beat GPT-3 in BoolQ, HellaSwag, WinoGrande, ARC-e and ARC-c benchmarks (not by much though). Examples that you've seen are likely to be based on some form quantisation / poor prompt that degrade output. My understanding that the only quantisation that doesn't seem to hurt the output is llm.int8 by Tim Dettmers. You should be able to run LLaMA 13B (8 bit quantised) on the 3090 or 4090 consumer grade GPU as of now. Also, you'd need a prompt such as LLaMA precise[2] in order to get ChatGPT like…

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by sitic

Mon, 20 Mar 2023 22:54:03 GMT

The LLaMA paper contradicts this view: "[...] Although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens." https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by aroo

Mon, 13 Mar 2023 00:34:52 GMT

The original LLaMA paper has some benchmarks. https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by ShamelessC

Sun, 12 Mar 2023 01:37:57 GMT

Here is the paper and synopsis: https://arxiv.org/abs/2302.13971 > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by v64

Sat, 11 Mar 2023 21:15:30 GMT

One finding in the LLaMA paper [1] is that our current large models are undertrained. LLaMA with 13B params outperforms GPT-3 175B (not ChatGPT), but an "instruct" version of LLaMA was finetuned over the 65B model and did quite well. [1] https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by doctoboggan

Sat, 11 Mar 2023 04:01:32 GMT

According to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf

Paper and discussion history. A mention is not an endorsement.

LLaMA: Open and Efficient Foundation Language Models — mentioned by zavrel

Mon, 06 Mar 2023 08:16:26 GMT

There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel https://arxiv.org/abs/2209.01188 Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971

Paper and discussion history. A mention is not an endorsement.