HackerPapers

arXiv:1906.02243cs.CL

Energy and Policy Considerations for Deep Learning in NLP

Emma Strubell, Ananya Ganesh, Andrew McCallum

Abstract

Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.

arXiv comments
In the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence, Italy. July 2019
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 8 identifiable commenters in 8 comments across 7 threads, from Sep 2019 to Feb 2024. Found in 4 HN submissions from Jun 2019 to Oct 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. I don't want anything your AI generates 1 comment

    • I'm not sure your premise even makes any sense here, because it doesn't take an artist much more resources to produce art then it took them to just exist for the same amount of time. They're still just eating, sleeping, making basic usage of the computer, using heating and…

      logicprogView on HNParent context

  2. The growing energy footprint of artificial intelligence ("AI") [pdf] 1 comment

    • Model training can at least follow the sun or clean energy forecasts. Latency from distance isn’t an issue as it would be for serving users requests. You can stage data for training runs. If you train during periods where the price of power goes negative, you get paid to take…

      toomuchtodoView on HNParent context

  3. Timnit Gebru on her sacking by Google, AI’s dangers and big tech’s biases 2 comments

    • The actual paper [0] that Gebru wrote referenced Strubell's climate impact paper [1]. None of them make the assumption that google compute runs from jet fuel. They compare training the models to flying, driving, etc. In the "Gender Shades" paper [2] as you call it, she doesn't…

      mempkoView on HNParent context

    • I don't know if you just... hope people don't read your citations. But as far as I could see from your citations you're not making sensible claims. The first citation the article talks about her citing someone else's work about carbon footprint. So firstly, they're not her…

      SilverBirchView on HNParent context

  4. Small Language Models Are Also Few-Shot Learners 1 comment

    • This is somewhat well covered in prior research, in so far as the information required is available. Here are some recent evaluations of the carbon footprint of modern models: https://arxiv.org/abs/1906.02243 (cited in the above paper) > The U.S. Environmental Protection Agency…

      soraki_soladeadView on HNParent context

  5. Deep Learning's Diminishing Returns 1 comment

    • Here's the original study[1] that seems to be the primary source for this article. It's an important study from a respectable journal. To be frank, it's pretty disconcerting that the top comments on this thread are those writing off the topic on the premise alone while those…

      culiView on HNParent context

  6. Green AI 1 comment

    • The citation for the "surprisingly large carbon footprint" [0] is crazy. The paper alleges that a car, including fuel, has a lifetime footprint of 126,000 lbs of CO₂e, and training a big NN transformer consumes 626,155 lbs of CO₂e, almost a 5-fold increase. [0]…

      cssView on HNParent context

  7. The U.S. economy is growing and using less and less stuff to do so 1 comment

    • On the other hand, as software and frameworks get more complex, they require more compute to run. Software hasn't decoupled economic growth from energy consumption, it has just made the energy consumption harder to see. Edit: since people are asking for examples, I would have to…

      brigaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.