Measuring the Algorithmic Efficiency of Neural Networks · HackerPapers discussions

Measuring the Algorithmic Efficiency of Neural Networks — mentioned by Strilanc

Fri, 04 Mar 2022 18:11:28 GMT

I'm a bit worried for that analog-matrix-multiplication-for-AI company. I vaguely remember reading somewhere that the past half century is littered with companies whose value proposition was "our specialized thing does 10x better than digital transistors" and then they predictably just got steamrolled by Moore's law. And although Dennard scaling ended two decades ago, the flops-per-watt and flops-per-second of AI-specialized chips like TPUs has been improving substantially recently [1][2]. 1: https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Third_g... 2: https://arxiv.org/abs/2005.04305

Paper and discussion history. A mention is not an endorsement.

Measuring the Algorithmic Efficiency of Neural Networks — mentioned by Isinlor

Sun, 06 Feb 2022 14:53:47 GMT

Roughly half/half, skewed more towards improvements in algorithms. Measuring the Algorithmic Efficiency of Neural Networks: https://arxiv.org/abs/2005.04305 In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months…

Paper and discussion history. A mention is not an endorsement.

Measuring the Algorithmic Efficiency of Neural Networks — HN submission

Thu, 10 Jun 2021 10:12:09 GMT

Measuring the Algorithmic Efficiency of Neural Networks

37 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Measuring the Algorithmic Efficiency of Neural Networks — HN submission

Sun, 24 May 2020 18:21:30 GMT

Measuring the Algorithmic Efficiency of Neural Networks

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.