HackerPapers

arXiv:1903.03129cs.DC

SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Beidi Chen, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava

Abstract

Deep Learning (DL) algorithms are the central focus of modern machine learning systems. As data volumes keep growing, it has become customary to train large neural networks with hundreds of millions of parameters to maintain enough capacity to memorize these volumes and obtain state-of-the-art accuracy. To get around the costly computations associated with large models and data, the community is increasingly investing in specialized hardware for model training. However, specialized hardware is expensive and hard to generalize to a multitude of tasks. The progress on the algorithmic front has failed to demonstrate a direct advantage over powerful hardware such as NVIDIA-V100 GPUs. This paper provides an exception. We propose SLIDE (Sub-LInear Deep learning Engine) that uniquely blends smart randomized algorithms, with multi-core parallelism and workload optimization. Using just a CPU, SLIDE drastically reduces the computations during both training and inference outperforming an optimized implementation of Tensorflow (TF) on the best available GPU. Our evaluations on industry-scale recommendation datasets, with large fully connected architectures, show that training with SLIDE on a 44 core CPU is more than 3.5 times (1 hour vs. 3.5 hours) faster than the same network trained using TF on Tesla V100 at any given accuracy level. On the same CPU hardware, SLIDE is over 10x faster than TF. We provide codes and scripts for reproducibility.

arXiv comments
Published at MLSys 2020
Journal reference
DOI
Subjects
cs.DCDistributed, Parallel, and Cluster Computing cs.LGMachine Learning

On Hacker News

Linked by 9 identifiable commenters in 11 comments across 9 threads, from Mar 2020 to Feb 2025. Found in 4 HN submissions from Mar 2019 to Jan 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. (Ab)using general search algorithms on dynamic optimization problems (2023) 1 comment

    • Related: a few years ago, various people were exploring locality sensitive hashing as an alternative to neural networks(SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems [1]). This never took off. I don't if it didn't work or…

      joe_the_userView on HNParent context

  2. Two sparsities are better than one: Performance of sparse-sparse networks 1 comment

    • > Sparse operations still don’t have good support in any of the generic SIMD instructions (avx2/512, Neon), or in GPU… This is incorrect. For GPUs: Nvidia A100 (launched in 2020) has sparse matrix multiplication support for inference. For CPUs: Check out SLIDE from 2019…

      rajnathaniView on HNParent context

  3. New features coming in Julia 1.7 1 comment
  4. pHash – An open source perceptual hash library (2013) 1 comment

    • Sparse neural networks can be optimized by implementing them using Locality Sensitive Hashing (2019 paper): https://arxiv.org/pdf/1903.03129.pdf Fun fact, last week one of the main authors of the above 2019 paper spun off a company for the above approach…

      rajnathaniView on HNParent context

  5. CPU-based algorithm trains deep neural nets up to 15 times faster than top GPU 3 comments

    • https://arxiv.org/abs/1903.03129 https://github.com/keroro824/HashingDeepLearning

      exegeteView on HNParent context

    • Paper: https://arxiv.org/abs/1903.03129 Code: https://github.com/keroro824/HashingDeepLearning Edit: Here is the new paper referred to https://arxiv.org/abs/2103.10891 Here is the new code: https://github.com/RUSH-LAB/SLIDE The paper I linked above was the first paper of which…

      exegeteView on HNParent context

    • It's not the first similar claim, there was one that large CPU's can beat GPUs by creating a very sparse matrix using spatial hashing. Preserving the gradient but finding it with vastly less computation by simply not multiplying elements that don't have much or any impact. EDIT…

      guywhocodesView on HNParent context

  6. Researchers propose faster, more efficient alternatives to backpropagation 1 comment

    • I found it very weird that the SLIDE algorithm from early 2019 isn’t mentioned. Maybe I missed it or maybe it is compared just deeper in the referenced publications? SLIDE seems way, way superior to any of the listed solutions or approaches, as far as I could tell on a first…

      mlthoughts2018View on HNParent context

  7. Deep Learning Breakthrough Sees One Xeon Destroying 8x Nvidia Tesla V100 GPUs 1 comment

    • No idea, it was probably sponsored by them but the concept should be the same for AMD and others too. EDIT: here's a bit more context (they've been contacted by Intel after publishing initial PoC) source: https://news.rice.edu/2020/03/02/deep-learning-rethink-overc... paper…

      cztomsikView on HNParent context

  8. Smart Algorithms beat Hardware Acceleration for Large-Scale Deep Learning 1 comment
  9. Slide: First algorithm for training deep neural nets faster on CPUs than GPUs 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.