HackerPapers

arXiv:2301.00774cs.LG

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Elias Frantar, Dan Alistarh

Abstract

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 9 comments across 9 threads, from Jan 2023 to Apr 2024. Found in 1 HN submission from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Show HN: Speeding up LLM inference 2x times (possibly) 1 comment

    • Essentially the algorithm is about pruning parameters on the fly and making the weight matrix sparse by setting least significant weights to zero, where "least significant" is defined in terms of the rank of the absolute value of the pruned weight within its group? A search for…

      renonceView on HNParent context

  2. Sam Altman goes before US Congress to propose licenses for building AI 1 comment

    • There's no chance that we've peeked from a bang for buck sense - we still haven't adequately investigated sparse networks. Relevantish: https://arxiv.org/abs/2301.00774 The fact that we can reach those levels of sparseness with pruning also indicates that we're not doing a very…

      yargView on HNParent context

  3. Run Llama 13B with a 6GB graphics card 1 comment

    • Training uses gradient descent, so you want to have good precision during that process. But once you have the overall structure of the network, https://arxiv.org/abs/2210.17323 (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy. It…

      sp332View on HNParent context

  4. OpenLLaMA: An Open Reproduction of LLaMA 1 comment
  5. GitHub Copilot emits GPL code 1 comment

    • While I agree we probably aren't getting exponentially increasing parameter counts (GPT4 is by all accounts 1T paramaters and of course, it is significantly better than GPT3) we are still seeing lots of improvements - 3.5 is much better than 3, based "just" on InstructGPT/RLHF…

      lhlView on HNParent context

  6. StableLM: A new open-source language model 1 comment

    • I think "sweet spot" is going to depend on your task, but here's a good recent paper that may give you some more context on thinking about training and model sizes: https://www.harmdevries.com/post/model-size-vs-compute-overh... There have also been quite a few developments on…

      lhlView on HNParent context

  7. Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa 1 comment

    • "Pruning" is the main term you are looking for, with a variety of methods to do so. https://blog.dataiku.com/making-neural-networks-smaller-for-... https://arxiv.org/abs/2301.00774 https://tivadardanka.com/blog/how-to-compress-a-neural-netwo... There's also quantization. It's…

      snerblesView on HNParent context

  8. Neural Chip Plays Doom Using a Thousandth of a Watt 1 comment

    • what percentage of weights could be thrown out post-training At least 60% according to this paper: https://arxiv.org/abs/2301.00774

      p1eskView on HNParent context

  9. Some Remarks on Large Language Models 1 comment

    • I think quantization (e.g. 4-bit, https://arxiv.org/abs/2212.09720) and sparsity (e.g. SparseGPT, https://arxiv.org/abs/2301.00774) will bring down inference cost. Edit: This isn’t handwaving btw, this is to say some fairly decent solutions are available now.

      woodsonView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.