SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot · HackerPapers discussions

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by renonce

Thu, 18 Apr 2024 04:50:03 GMT

Essentially the algorithm is about pruning parameters on the fly and making the weight matrix sparse by setting least significant weights to zero, where "least significant" is defined in terms of the rank of the absolute value of the pruned weight within its group? A search for model pruning turns out many results, including https://arxiv.org/abs/2305.11627 which discusses "magnitude pruning" as a baseline, and refers to https://arxiv.org/pdf/2301.00774.pdf, which asserts in the introduction: > First, as shown in Figure 1, SparseGPT can induce uniform layerwise sparsity of up to 60% in e.g.…

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by yarg

Tue, 16 May 2023 23:05:50 GMT

There's no chance that we've peeked from a bang for buck sense - we still haven't adequately investigated sparse networks. Relevantish: https://arxiv.org/abs/2301.00774 The fact that we can reach those levels of sparseness with pruning also indicates that we're not doing a very good job of generating the initial network conditions. Being able to come up with trainable initial settings for sparse networks across different topologies is hard, but given that we've had a degree of success with pre-trained networks, pre-training and pre-pruning might also allow for sparse networks with minimally…

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by sp332

Sun, 14 May 2023 22:45:39 GMT

Training uses gradient descent, so you want to have good precision during that process. But once you have the overall structure of the network, https://arxiv.org/abs/2210.17323 (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy. It seems you can cut down further for larger models. For the 13B Llama-based ones, going below 5 bits per parameter is noticeably worse, but for 30B models you can do 4 bits. The same group did another paper https://arxiv.org/abs/2301.00774 which shows that in addition to reducing the precision of each parameter, you can…

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — HN submission

Wed, 03 May 2023 16:44:19 GMT

SparseGPT: Language Models Can Be Accurately Pruned in One-Shot

211 points; 62 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by sebzim4500

Wed, 03 May 2023 10:04:18 GMT

https://arxiv.org/pdf/2301.00774.pdf

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by lhl

Fri, 21 Apr 2023 22:37:52 GMT

While I agree we probably aren't getting exponentially increasing parameter counts (GPT4 is by all accounts 1T paramaters and of course, it is significantly better than GPT3) we are still seeing lots of improvements - 3.5 is much better than 3, based "just" on InstructGPT/RLHF training. Models are getting better as well - LLaMA 30B beats/matches GPT-3 on raw eval benchmarks at 1/6 the parameter count. We're also seeing lots of optimizations with new models (RoPE/RoPER embedding, Swish/GeLU activation, Flash Attention, etc) but I think some the most interesting gains we'll be seeing soon is…

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by lhl

Wed, 19 Apr 2023 19:05:24 GMT

I think "sweet spot" is going to depend on your task, but here's a good recent paper that may give you some more context on thinking about training and model sizes: https://www.harmdevries.com/post/model-size-vs-compute-overh... There have also been quite a few developments on sparsity lately. Here's a technique SparseGPT which suggests that you can prune 50% of parameters with almost no loss in performance for example: https://arxiv.org/abs/2301.00774

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by snerbles

Wed, 29 Mar 2023 00:36:51 GMT

"Pruning" is the main term you are looking for, with a variety of methods to do so. https://blog.dataiku.com/making-neural-networks-smaller-for-... https://arxiv.org/abs/2301.00774 https://tivadardanka.com/blog/how-to-compress-a-neural-netwo... There's also quantization. It's currently where a lot of the current grassroots research is happening, if you've seen the llama.cpp repo posted here on HN, it uses the original 16-bit float weights downsized to 4-bit integers with a comparable reduction in resource usage with relatively little performance loss. The big players are also using…

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by p1esk

Sat, 04 Mar 2023 07:10:21 GMT

what percentage of weights could be thrown out post-training At least 60% according to this paper: https://arxiv.org/abs/2301.00774

Paper and discussion history. A mention is not an endorsement.

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot — mentioned by woodson

Tue, 03 Jan 2023 19:15:23 GMT

I think quantization (e.g. 4-bit, https://arxiv.org/abs/2212.09720) and sparsity (e.g. SparseGPT, https://arxiv.org/abs/2301.00774) will bring down inference cost. Edit: This isn’t handwaving btw, this is to say some fairly decent solutions are available now.

Paper and discussion history. A mention is not an endorsement.