arXiv:1404.5997cs.NE
I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.
Linked by 11 identifiable commenters in 12 comments across 12 threads, from Apr 2014 to Jun 2022. Found in 1 HN submission from Jul 2014 to Jul 2014. Open submission discussions.
Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: https://arxiv.org/abs/1404.5997 When attempting to train transformers, I do notice a lot of time spend on allreduce more than with CNN models, probably due the parameter sizes. OWT…
We embrace that as well: https://arxiv.org/abs/1404.5997
It's been this way since day 1. NVLINK remains the only real Tesla differentiator (although mini NVLINK is available on the new Turing consumer GPUs so WTFever). But because none of the DL frameworks support intra-layer model parallelism, all of the networks we see tend to run…
Oh, I agree 100%. But the truth is that without the hardware vendors putting significant resources into OpenCL it just isn't competitive and won't be until that happens. The truth is that most of the work in Deep Learning is developing new NN architectures and other algorithmic…
http://arxiv.org/abs/1404.5997
Fair enough, but when neither Intel nor AMD is bothering to put out competitive software/hardware solutions for deep learning, which is apparently going to be a ~$40B market by 2020, I predict continued dominance here. That said, since AMD is getting beaten by a factor of 2-3 in…
One weird trick for parallelizing ConvNets: http://arxiv.org/abs/1404.5997 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks: http://arxiv.org/abs/1511.06434 A Neural Network of Artistic Style: http://arxiv.org/abs/1508.06576
Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of…
How is this on the front page? This is a completely incoherent. For anyone actually interested in some interesting techniques for multi-GPU DNN training, http://arxiv.org/pdf/1404.5997v2.pdf and references therein are probably a good start.
Hey Dave! (Craig here) nice, achieved 6x speed up on an 8 gpu box (http://arxiv.org/pdf/1404.5997v2.pdf). Thanks for the pointer and hope to see you soon.
A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1] By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take…
Connected through shared HN threads, not inferred similarity or a recommendation.
1. I missed this paper https://www.microsoft.com/en-us/research/uploads/prod/2018/0... when doing my own feature map compression research: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de.... Thanks for point it out! 2. Otherwise, most of these are obvious…
liuliuView on HNParent context