Llama 2: Open Foundation and Fine-Tuned Chat Models · HackerPapers discussions

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by like_any_other

Wed, 24 Jun 2026 19:50:06 GMT

Unsurprising result - here's Microsoft (in collaboration with MIT, Carnegie Mellon, and University of Washington) on how they measure & censor hate speech in their AIs [0]: Our ultimate aim is to shift power dynamics to targets of oppression. Therefore, we do not consider identity dimensions that are historically the agents of oppression (e.g., whiteness, heterosexuality, able-bodied-ness). As promised, their safety scores exclude Whites [1], and their training data [2,3] labels the following as hate: - "stop hurting white people" [no other results for "stop hurting"] [2] - "i dont want to be…

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by like_any_other

Sat, 19 Apr 2025 14:50:42 GMT

Microsoft (in collaboration with MIT, Carnegie Mellon, and University of Washington) on how they measure & censor hate speech in their AIs [0]: Our ultimate aim is to shift power dynamics to targets of oppression. Therefore, we do not consider identity dimensions that are historically the agents of oppression (e.g., whiteness, heterosexuality, able-bodied-ness). Their training data [1,2] labels the following as hate: - "stop hurting white people" [no other results for "stop hurting"] - "i dont want to be a minority in my own country." - "White Americans and White Europeons have too few…

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by rafram

Thu, 06 Mar 2025 16:11:56 GMT

Right, the original LLaMA paper. LLaMA 2 and 3 are significantly more capable models trained on orders of magnitude more data, and those papers notably do not say where the data comes from. The LLaMA 3 paper helpfully mentions that "Much of the data we utilize is obtained from the web," so I guess that's better than nothing! LLaMA 2: https://arxiv.org/pdf/2307.09288 LLaMA 3: https://arxiv.org/pdf/2407.21783

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by ninjin

Tue, 25 Jun 2024 11:07:10 GMT

Facebook has been touting their work as both "open source" and "open science", despite being in direct violation of the terms [1, 2]. Others have done similar things, but not on such a big scale and with models with such big reach. In the community, the terms have now become sufficiently muddied that it is hard to have a conversation about what an open model even is. [1]: https://arxiv.org/abs/2307.09288 [2]: https://opensource.org/blog/metas-llama-2-license-is-not-ope... Thankfully, OLMo [3] reverted their decision to go with a license that contained usage restrictions and hopefully this…

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by michaelt

Thu, 13 Jun 2024 09:43:50 GMT

Then, you should check out papers like https://arxiv.org/abs/2302.13971 and https://arxiv.org/abs/2307.09288 In the paper covering the original Llama they explicitly list their data sources in table 1 - including saying that they pretrained on the somewhat controversial books3 dataset. The paper for Llama 2 also explicitly says they don't take data from Meta's products and services; and that they filter out data from sites known to contain a lot of PII. Although it is more coy about precisely what data sources they used, like many such papers are.

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by hapanin

Fri, 15 Dec 2023 15:59:24 GMT

Since nobody is actually recommending papers, here's an incomplete reading list that I sent out to some masters students I work with so they can understand the current research (academic) my little team is doing: Paper reference / main takeaways / link instructGPT / main concepts of instruction tuning / https://proceedings.neurips.cc/paper_files/paper/2022/hash/b... self-instruct / bootstrap off models own generations / https://arxiv.org/pdf/2212.10560.pdf Alpaca / how alpaca was trained / https://crfm.stanford.edu/2023/03/13/alpaca.html Llama 2 / probably the best chat model we can train on…

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by tydunn

Tue, 31 Oct 2023 01:26:30 GMT

This is a lot of tokens. Llama 2 was trained on two trillion tokens [1] [1] https://arxiv.org/abs/2307.09288

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — mentioned by hexaga

Sun, 08 Oct 2023 00:19:20 GMT

Ah, I see. You are unfamiliar with how SOTA language models are actually trained in current day, what RLHF is in detail, and how it differs from other training phases. I'd encourage you to read through the GPT3 [0] and llama2 [1] papers for more salient descriptions of the specific mechanics involved and why RLHF is used at all, before trying to argue from some kind of introduction to ml 101 level simplification of how SOTA models are produced. - [0]: https://arxiv.org/abs/2005.14165 - [1]: https://arxiv.org/abs/2307.09288 For example, from the llama2 paper, section "3.2 Reinforcement…

Paper and discussion history. A mention is not an endorsement.

Llama 2: Open Foundation and Fine-Tuned Chat Models — HN submission

Sat, 29 Jul 2023 07:37:27 GMT

Llama 2: Open Foundation and Fine-Tuned Chat Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.