How Can We Be So Dense? The Benefits of Using Highly Sparse Representations — HN submission

Wed, 07 Oct 2020 22:15:41 GMT

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations — mentioned by p1esk

Mon, 19 Aug 2019 20:23:38 GMT

Good point. I think it's a safe bet to focus on dense dot product in hardware for the foreseeable future. However, to their defense: 1. It's not clear that supporting sparse operands in hw would result in significant overhead. 2. DL models are still pretty sparse (I bet even those models with GELU still have lots of very small values that could be safely rounded to zero). 3. Sparsity might have some benefits (e.g. https://arxiv.org/abs/1903.11257).

Paper and discussion history. A mention is not an endorsement.

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations · HackerPapers discussions

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations — HN submission

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations — mentioned by p1esk