HackerPapers

arXiv:1706.03301cs.LG

Neural networks and rational functions

Matus Telgarsky

Abstract

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree $O(\text{polylog}(1/ε))$ which is $ε$-close, and similarly for any rational function there exists a ReLU network of size $O(\text{polylog}(1/ε))$ which is $ε$-close. By contrast, polynomials need degree $Ω(\text{poly}(1/ε))$ to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.

arXiv comments
To appear, ICML 2017
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from May 2018 to Jun 2018.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Polynomial Regression as an Alternative to Neural Nets 1 comment

    • I would be less surprised if the paper claimed rational functions instead of polynomials. NN's are ~general non-linear functions which polynomials are very poor at representing (e.g. you can't encode asymptotic behavior in polynomials). edit: here's a paper about rational…

      cozzydView on HNParent context

  2. Neural Networks and Primes 1 comment

    • An interesting idea. A paper from last year showed that neural networks and rational functions efficiently approximate one another [1]. [1] https://arxiv.org/abs/1706.03301

      wjn0View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.