Petals: Collaborative Inference and Fine-tuning of Large Models · HackerPapers discussions

Petals: Collaborative Inference and Fine-tuning of Large Models — HN submission

Sun, 19 Jul 2026 15:37:21 GMT

Petals: Collaborative Inference and Fine-Tuning of Large Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by ftxbro

Mon, 20 Mar 2023 23:28:44 GMT

This link just goes to their website. Last I looked at this project, I was happy that it existed but I was disappointed (given my over-optimistic expectations) for two reasons: 1) It's for the BLOOM model which isn't great compared to somewhat recent gpts. Like I think I read that it's worse than the openai models on a per parameter basis. 2) It's faster than using RAM/SSD as faux VRAM but 'only' by 10x. That was even before LLaMA or its improvements had existed for running locally. So by my old understanding, bloom/petals wouldn't be even as good as those ones even though it technically has…

Paper and discussion history. A mention is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — HN submission

Fri, 17 Mar 2023 07:40:54 GMT

Petals: Collaborative Inference and Fine-Tuning of Large Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by zavrel

Mon, 06 Mar 2023 08:16:26 GMT

There was great work last year on distributed training and inference: Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel https://arxiv.org/abs/2209.01188 Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971

Paper and discussion history. A mention is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by borzunov

Tue, 21 Feb 2023 00:46:50 GMT

Note that the authors report the speed of generating many sequences in parallel (per token): > The batch size is tuned to a value that maximizes the generation throughput for each system. > FlexGen cannot achieve its best throughput in [...] single-batch case. For 175B models, this likely means that the system takes a few seconds for each generation step, but you can generate multiple sequences in parallel and get a good performance _per token_. However, what you actually need for ChatGPT and interactive LM apps is to generate _one_ sequence reasonably quickly (so it takes <= 1 sec/token to…

Paper and discussion history. A mention is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by joaogui1

Sun, 15 Jan 2023 21:08:39 GMT

This initiative is getting some pretty good practical results https://arxiv.org/abs/2209.01188, taking a look at their citations should lead you to other examples in the field

Paper and discussion history. A mention is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by borzunov

Mon, 02 Jan 2023 19:15:26 GMT

Theoretical best-case for RAM offloading is 5.5 sec/token, for SSD offloading - 22 sec/token. Implementations we've tested are not faster than 10 sec/token though. See details in our paper: https://arxiv.org/pdf/2209.01188.pdf

Paper and discussion history. A mention is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by taink

Mon, 02 Jan 2023 10:00:50 GMT

It's mentioned in their paper: https://arxiv.org/pdf/2209.01188.pdf Several recent works aim to democratize LLMs by “offloading” model parameters to slower but cheaper memory (RAM or SSD), then running them on the accelerator layer by layer (Pudipeddi et al., 2020; Ren et al., 2021). This method allows running LLMs with a single low-end accelerator by loading parameters from RAM justin-time for each forward pass. Offloading can be efficient for processing many tokens in parallel, but it has inher- ently high latency: for example, generating one to- ken with BLOOM-176B takes at least 5.5…

Paper and discussion history. A mention is not an endorsement.

Petals: Collaborative Inference and Fine-tuning of Large Models — mentioned by borzunov

Tue, 20 Dec 2022 19:50:32 GMT

See technical details in the paper: https://arxiv.org/abs/2209.01188

Paper and discussion history. A mention is not an endorsement.