Mad Max: Affine Spline Insights into Deep Learning · HackerPapers discussions

Mad Max: Affine Spline Insights into Deep Learning — mentioned by magicalhippo

Mon, 08 Jul 2024 11:04:10 GMT

As I understand it, the regions are simply the pieces that constitute the partitioning of the input domain, ie vector space formed by the the weights. There's some more details in one of the referenced papers[1], section 3.1 and onward. The argument in that paper is that the layers in a typical deep neural network partitions the input domain into regions, where each region has its own affine mapping of the input. For any arbitrary activation function, one would have to find the partitioning as well as the per-region parameters of the affine mappings. However since all the common activation…

Paper and discussion history. A mention is not an endorsement.

Mad Max: Affine Spline Insights into Deep Learning — mentioned by magicalhippo

Mon, 08 Jul 2024 01:04:08 GMT

A follow-up comment after having studied the paper a bit more, since you asked about where the geometry comes into play. One of the references the paper provide is to this[1] paper, which shows how the non-linear layers in modern deep neural networks partitions the input into regions and applies region-dependent affine mappings[2] to generate the output. It also mentions how that connects to vector quantization and k-means clustering. So, the geometric perspective isn't referring to your typical high-school geometry, but more abstract concepts like vector spaces[3] and combinatiorial…

Paper and discussion history. A mention is not an endorsement.