Multiplying Matrices Without Multiplying · HackerPapers discussions

Multiplying Matrices Without Multiplying — mentioned by nickpsecurity

Tue, 28 May 2024 23:16:32 GMT

There's work on replacing multiplication. Here's four examples: https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_... https://arxiv.org/abs/2012.03458 https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/E... https://arxiv.org/pdf/2106.10860

Paper and discussion history. A mention is not an endorsement.

Multiplying Matrices Without Multiplying — mentioned by joennlae

Tue, 21 Nov 2023 09:35:56 GMT

Author here: Thank you for the feedback :-) A lot of the work regarding the comparison with „simple“ approximate matrix multiplication has been done in the preceding paper: https://arxiv.org/abs/2106.10860 While I share your enthusiasm regarding the potential, we have to be careful about the limiting factors. Our main contributions on the algorithmic side are the reformulation of Maddness such that it is differentiable (autogradable), and we can use it in e2e DNN training, as decision trees are not differentiable. We are still in the process of understanding how to optimise the training. In…

Paper and discussion history. A mention is not an endorsement.

Multiplying Matrices Without Multiplying — mentioned by ffast-math

Sun, 19 Jun 2022 04:16:42 GMT

We found sparse, truncated PCA to be the most competitive baseline. We beat it by a lot (see the paper [1]), but the other big drawback is that trading off the rank vs sparsity was an ugly hyperparameter tuning problem. By ugly, I mean that the results were really sensitive to getting this right, it wasn't easy to set a priori, and took a while to iterate on because the sparse PCA trained much more slowly than any other practical alternative. There are situations where PCA/SVD is the right approach though. Namely, if you need really little error, our method often can't do that, whereas…

Paper and discussion history. A mention is not an endorsement.

Multiplying Matrices Without Multiplying — mentioned by ffast-math

Sun, 19 Jun 2022 03:51:05 GMT

There's definitely a tradeoff between speed and accuracy. We characterize this for various problems in the paper (https://arxiv.org/pdf/2106.10860.pdf), but tl;dr is that it speeds things up more at a given level of error when there's more redundancy in your matrices. Back-of-the-envelope calculation suggests that this won't beat tensor cores on NVIDIA GPUs. This is basically because ~half the die is an ASIC for dense (and 2:4 sparse) matmuls, with no support for the sparsity structure we induce. If 1:16 sparsity were supported or there were a batched warp_shuffle instruction, we'd get…

Paper and discussion history. A mention is not an endorsement.

Multiplying Matrices Without Multiplying — mentioned by cgreerrun

Sat, 18 Jun 2022 19:38:07 GMT

Maddness is their more recent work and yields 100x speedups: https://arxiv.org/pdf/2106.10860.pdf The code for Maddness is in the same github repo if you search for "Mithral". SIMD instructions can work wonders in the right context.

Paper and discussion history. A mention is not an endorsement.

Multiplying Matrices Without Multiplying — HN submission

Wed, 01 Sep 2021 00:02:33 GMT

Multiplying Matrices Without Multiplying

235 points; 122 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Multiplying Matrices Without Multiplying — HN submission

Wed, 23 Jun 2021 15:19:31 GMT

Multiplying Matrices Without Multiplying

4 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.