Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks — mentioned by yorwba

Mon, 30 Oct 2017 05:29:15 GMT

They cite https://arxiv.org/abs/1610.09887 for their definition of network depth, which defines it in such a way that e.g. a ReLU network of depth 2 is of the form linear2(ReLu(linear1(input))). That means, depth is the number of linear layers. The "depth-three" model in this paper is a bit strange in that their second layer has only one output, so the third linear layer doesn't have any effect. I would have called this "depth two"; but it is internally consistent with their definition of depth. > How is ReLU smooth? It is 1-Lipschitz, which is smooth enough for them.

Paper and discussion history. A mention is not an endorsement.

Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks · HackerPapers discussions

Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks — mentioned by yorwba