Fast Algorithms for Convolutional Neural Networks · HackerPapers discussions

Fast Algorithms for Convolutional Neural Networks — mentioned by bwasti

Mon, 02 Sep 2019 04:04:43 GMT

Note that this a layout trick and not an algorithmic one. An algorithmic speed up that is good for dense convolutions with small kernels is to use Winograd: https://arxiv.org/abs/1509.09308 For large kernels, implementing an FFT tends to help. Also worth keeping in mind that many modern networks use depthwise separable convolutions, which are channel wise convolutions (skipping a reduction over the channels, which is a memory bound operation) followed by 1x1 convolutions (which are exactly matrix multiplications with no im2col step).

Paper and discussion history. A mention is not an endorsement.

Fast Algorithms for Convolutional Neural Networks — mentioned by peterhj

Wed, 05 Jun 2019 23:17:11 GMT

One counterexample: https://arxiv.org/abs/1509.09308v1

Paper and discussion history. A mention is not an endorsement.

Fast Algorithms for Convolutional Neural Networks — mentioned by johndough

Thu, 09 May 2019 19:33:44 GMT

The answer is: No one really knows because cuBLAS is closed source. But to get within the same order of magnitude, tiling the workload for better cache utilization is usually the most important step. This article [1] explains it quite well and also lists a few other tricks. In addition, there's also the fast Fourier transform for large filter kernels and Winograd convolutions [2] for small filter kernels. [1] https://cnugteren.github.io/tutorial/pages/page1.html [2] https://arxiv.org/pdf/1509.09308.pdf

Paper and discussion history. A mention is not an endorsement.