One weird trick for parallelizing convolutional neural networks · HackerPapers discussions

One weird trick for parallelizing convolutional neural networks — mentioned by liuliu

Thu, 09 Jun 2022 21:24:01 GMT

1. I missed this paper https://www.microsoft.com/en-us/research/uploads/prod/2018/0... when doing my own feature map compression research: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de.... Thanks for point it out! 2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick"…

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by liuliu

Mon, 10 Feb 2020 21:49:28 GMT

Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: https://arxiv.org/abs/1404.5997 When attempting to train transformers, I do notice a lot of time spend on allreduce more than with CNN models, probably due the parameter sizes. OWT seems to be natural to exploit for this situation (a lot of GEMMs, lot time spent on allreduce). Edit: Read the paper. The implementation is much less tricky than OWT, but for a good reason probably. Language model's GEMMs are smaller, therefore, partition the model would have efficiency impact (smaller GEMM will be…

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by backpropaganda

Mon, 04 Feb 2019 01:14:43 GMT

We embrace that as well: https://arxiv.org/abs/1404.5997

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by oneshot908

Tue, 25 Sep 2018 14:36:34 GMT

It's been this way since day 1. NVLINK remains the only real Tesla differentiator (although mini NVLINK is available on the new Turing consumer GPUs so WTFever). But because none of the DL frameworks support intra-layer model parallelism, all of the networks we see tend to run efficiently in data parallel because doing anything else makes them communication-limited, which they aren't because data scientists end up building networks that aren't, chicken and the egg style. I continue to be boggled that Alex Krizhevsky's One Weird Trick never made it to TensorFlow or anywhere else…

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by nl

Fri, 22 Jul 2016 15:40:23 GMT

Oh, I agree 100%. But the truth is that without the hardware vendors putting significant resources into OpenCL it just isn't competitive and won't be until that happens. The truth is that most of the work in Deep Learning is developing new NN architectures and other algorithmic optimisations. If you are working in the field there is no reason to put up with second class support from non NVidia vendors - just build in TensorFlow, Torch or a couple of other frameworks and wait for the day (one day, we are promised!) when OpenCL is competitive. Then the framework backends get ported, your code…

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by mac01021

Thu, 26 May 2016 12:19:58 GMT

http://arxiv.org/abs/1404.5997

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by varelse

Sat, 07 May 2016 18:33:14 GMT

Fair enough, but when neither Intel nor AMD is bothering to put out competitive software/hardware solutions for deep learning, which is apparently going to be a ~$40B market by 2020, I predict continued dominance here. That said, since AMD is getting beaten by a factor of 2-3 in performance while Intel's Accelerator/FPGA solutions are getting beaten by a factor of 5-10, AMD has a real opportunity to be the competitor to NVIDIA here. And despite questions about it, GTX 1080 will be a fine deep learning board, especially if the framework engineers get off their lazy butts and implement Alex…

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by temuze

Wed, 23 Dec 2015 19:05:12 GMT

One weird trick for parallelizing ConvNets: http://arxiv.org/abs/1404.5997 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks: http://arxiv.org/abs/1511.06434 A Neural Network of Artistic Style: http://arxiv.org/abs/1508.06576

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by prajit

Mon, 09 Nov 2015 15:08:42 GMT

Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by ajtulloch

Tue, 25 Nov 2014 04:08:18 GMT

How is this on the front page? This is a completely incoherent. For anyone actually interested in some interesting techniques for multi-GPU DNN training, http://arxiv.org/pdf/1404.5997v2.pdf and references therein are probably a good start.

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by cr4zy

Tue, 19 Aug 2014 16:00:26 GMT

Hey Dave! (Craig here) nice, achieved 6x speed up on an 8 gpu box (http://arxiv.org/pdf/1404.5997v2.pdf). Thanks for the pointer and hope to see you soon.

Paper and discussion history. A mention is not an endorsement.

One weird trick for parallelizing convolutional neural networks — HN submission

Thu, 24 Jul 2014 05:02:01 GMT

One weird trick for parallelizing convolutional neural networks

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

One weird trick for parallelizing convolutional neural networks — mentioned by kastnerkyle

Mon, 28 Apr 2014 03:17:30 GMT

A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1] By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take advantage of layer properties to optimize the sharing between GPUs. Whether that reduction in data movement is enough to make it feasible for AWS/local LAN links is up for debate. [1] http://arxiv.org/abs/1404.5997

Paper and discussion history. A mention is not an endorsement.