LLaMA: Open and Efficient Foundation Language Models

I don't understand why it's even a question that Meta trained their LLM on copyrighted material. They say so in their paper! Quoting from their LLaMMa paper [Touvron et al., 2023]: > We include two book corpora in our training dataset: the Gutenberg Project, [...], and the…

JW_00000Feb 7, 2025View on HNParent context

Then, you should check out papers like https://arxiv.org/abs/2302.13971 and https://arxiv.org/abs/2307.09288 In the paper covering the original Llama they explicitly list their data sources in table 1 - including saying that they pretrained on the somewhat controversial books3…

michaeltJun 13, 2024View on HNParent context

The most detailed answer to that I've seen is the original LLaMA paper, which described exactly what that model was trained on (including lots of scraped copyrighted data) https://arxiv.org/abs/2302.13971 Llama 2 was much more opaque about the training data, presumably because…

simonwMar 27, 2024View on HNParent context

Because the training data/model size/compute tradeoff derived from that paper is highly suboptimal (too many parameters) compared to the ones from the later Deepmind scaling laws [1]. And then Meta researchers recommended using even smaller models, to trade-off training- and…

versteegenDec 17, 2023View on HNParent context

Facebook has been open-sourcing AI research longer than OpenAI has even had the concept of an "investor narrative". I struggle to understand how someone could jump to the conclusion of this being a "scorched earth" maneuver with so many other reasonable explanations. Facebook…

smoldesuSep 27, 2023View on HNParent context

Llama off the top of my head: https://arxiv.org/pdf/2302.13971.pdf

fpgaminerSep 6, 2023View on HNParent context

Llama: https://arxiv.org/pdf/2302.13971.pdf You can clearly see in table on second page that higher quality data is trained on more than 1 epoch. Most Open LLM's do this.

computerexSep 6, 2023View on HNParent context

The Llama 1 paper [1] was one of the earlier models to question the assumption that more params = better model. Since then they've released Llama 2 and this post is offering more evidence that reinforces their hypothesis. I wouldn't say it was an oversight by other labs that…

v64Sep 4, 2023View on HNParent context

Are we all reading the same complaint? They say: > in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an…

edentJul 9, 2023View on HNParent context

It's in the LLaMa.cpp original research paper. Tis mentioned in the brief. The research paper basically stated it was trained on Bibliotik, and other Internet "shadow library" corpuses. See the reference to the Gao et al, in the linked paper from the article. Paper linked in…

salawatJul 10, 2023View on HNParent context

It is very expensive to train these base models so a smaller size is more practical if you aren’t a big company with hundreds of powerful GPUs at hand. Table 15 from LLaMA paper[1] has some insightful figures: it took 135,168 GPU hours to train the 13B version and a bit more…

pythuxJun 18, 2023View on HNParent context

I don't think Facebooks moves were heavily strategically thought out. Instead, I think a small team of a few researchers made LLaMA and published a paper about it[1], and then realised to get any citations for their paper they'd need to let other researchers have the weights.…

londons_exploreJun 8, 2023View on HNParent context

I love bashing Intel as much as the other person, but looking at the number of tokens LLaMA[1] was trained on (1,400B tokens), the number of parameters of GPT-3[2] vs. the number of tokens it was trained on (175B params vs. 300B), it's not like the announced 1,000B param model…

ppaattrriicckkMay 22, 2023View on HNParent context

First link: https://github.com/ggerganov/llama.cpp Which in turn has the following as the first link: https://arxiv.org/abs/2302.13971 Is it really quicker to ask here than just browse content for a bit, skimming some text or even using Google for one minute?

capablewebMay 14, 2023View on HNParent context

They are likely doing some interpolation for 200B or benchmarking it in wrong way. e.g. Hellaswag accuracy for llama 7b is 0.76[1], but it is written 0.56 in the repo. Even at 200B tokens, it is higher than 0.56 for llama looking at the charts. [1]…

YetAnotherNickMay 3, 2023View on HNParent context

https://arxiv.org/pdf/2302.13971.pdf table 15. 1770394 A100-80GB hours to train the entire model suite at the going rate for cloud 8xA100-80GBs (~$12/hr if you could actually get capacity) is ~$2.6M, under extremely optimistic assumptions. YMMV on bulk pricing ;) "the more you…

eizApr 17, 2023View on HNParent context

Page 4 https://arxiv.org/abs/2302.13971 When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM. This means that training over our dataset containing 1.4T tokens takes approximately 21 days. At $4/GPU-hour per A100 80GB…

sp332Apr 17, 2023View on HNParent context

LLama 65B used 2048 80G A100s for 21 days: > When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM.[1] Note that you probably need to budget for double to triple that because things go wrong and it usually takes…

nlApr 17, 2023View on HNParent context

> ChatGPT competitors of EU origin? ChatGPT is not research. If you want real research: https://ai.facebook.com/blog/large-language-model-llama-meta... Edit: end paper is https://arxiv.org/abs/2302.13971 > Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet…

frozencellApr 13, 2023View on HNParent context

It's worth pointing out that size isn't everything. From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals. Those interested in doing more comparisons can look at…

lhlApr 10, 2023View on HNParent context

Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0]. Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right? [0] https://arxiv.org/pdf/2302.13971.pdf

yregApr 4, 2023View on HNParent context

https://arxiv.org/abs/2302.13971

dchestMar 31, 2023View on HNParent context

Check out the graph on page 3 of this PDF: https://arxiv.org/abs/2302.13971 The 33B model started beating the 7B when it had been trained on only 1/3 as much data. And then they kept training it to 40% more than the total that 7B was trained on. It's better.

sp332Apr 1, 2023View on HNParent context

I posted the wrong link sorry! https://arxiv.org/pdf/2302.13971.pdf (the Llama paper), table 7.

nlMar 30, 2023View on HNParent context

The stats shown are pretty compelling that this transformer + RNN approach works. Looking back at the Llama paper (https://arxiv.org/pdf/2302.13971.pdf), many of the results are comparable between llama and RWKV. E.g. Llama 13bn scores 80.1 on PIQA and RWKV scores 77.5 I wonder…

imranqMar 30, 2023View on HNParent context

The LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included.…

JW_00000Mar 29, 2023View on HNParent context

And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1

corfordMar 29, 2023View on HNParent context

Is there a benchmark comparing the two that I missed? Edit: The huggingface page has 0-shot benchmarks which you can compare against the llama paper https://huggingface.co/cerebras/Cerebras-GPT-13B https://arxiv.org/pdf/2302.13971.pdf

gpmMar 28, 2023View on HNParent context

There are benchmarks in the original LLaMA paper[1]. Specifically, on page 4 LLaMA 13B seems to beat GPT-3 in BoolQ, HellaSwag, WinoGrande, ARC-e and ARC-c benchmarks (not by much though). Examples that you've seen are likely to be based on some form quantisation / poor prompt…

rnosovMar 24, 2023View on HNParent context

The LLaMA paper contradicts this view: "[...] Although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens." https://arxiv.org/pdf/2302.13971.pdf

siticMar 20, 2023View on HNParent context

The original LLaMA paper has some benchmarks. https://arxiv.org/pdf/2302.13971.pdf

arooMar 13, 2023View on HNParent context

One finding in the LLaMA paper [1] is that our current large models are undertrained. LLaMA with 13B params outperforms GPT-3 175B (not ChatGPT), but an "instruct" version of LLaMA was finetuned over the 65B model and did quite well. [1] https://arxiv.org/pdf/2302.13971.pdf

v64Mar 11, 2023View on HNParent context

Here is the paper and synopsis: https://arxiv.org/abs/2302.13971 > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using…

ShamelessCMar 12, 2023View on HNParent context

According to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf

doctobogganMar 11, 2023View on HNParent context

There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel…

zavrelMar 6, 2023View on HNParent context