GPT-NeoX-20B: An Open-Source Autoregressive Language Model · HackerPapers discussions

GPT-NeoX-20B: An Open-Source Autoregressive Language Model — mentioned by kir-gadjello

Mon, 13 Mar 2023 20:53:10 GMT

If you are interested in the infrastructure-level details of how similar models are trained by lesser known groups, take a look at this paper: https://arxiv.org/abs/2204.06745 Quotes from the paper: Our model is trained using a codebase that builds on Megatron (Shoeybi et al., 2020) and DeepSpeed (Rasley et al., 2020) to facilitate efficient and straightforward training of large language models with tens of billions of parameters. We use the official PyTorch v1.10.0 release binary package compiled with CUDA 11.1. This package is bundled with NCCL 2.10.3 for distributed communications. We…

Paper and discussion history. A mention is not an endorsement.

GPT-NeoX-20B: An Open-Source Autoregressive Language Model — mentioned by cottonseed

Fri, 09 Dec 2022 22:46:17 GMT

Yes. EleutherAI is doing it, probably one of many: https://www.eleuther.ai/projects/gpt-neox/ https://github.com/EleutherAI/gpt-neox https://arxiv.org/abs/2204.06745 They have a 20B parameter model. I think the primary dataset for these open models is The Pile: https://arxiv.org/abs/2101.00027 (web scrape, pubmed, arxiv, github, wikipedia, etc. There is a nice diagram on page 2 that summarizes the contents.)

Paper and discussion history. A mention is not an endorsement.

GPT-NeoX-20B: An Open-Source Autoregressive Language Model — HN submission

Wed, 27 Apr 2022 04:27:21 GMT

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.