HackerPapers

arXiv:1806.06850cs.LG

Polynomial Regression As an Alternative to Neural Nets

Xi Cheng, Bohdan Khomtchouk, Norman Matloff, Pete Mohanty

Abstract

Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.

arXiv comments
23 pages, 1 figure, 13 tables
Journal reference
DOI
Subjects
cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Aug 2019 to May 2026. Found in 3 HN submissions from Jun 2018 to Sep 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. A polynomial autoencoder beats PCA on transformer embeddings 1 comment

    • There's also work in polynomial regression as an alternative to regular, neural networks. Polynomial Regression As an Alternative to Neural Nets (2018) https://arxiv.org/abs/1806.06850 Π-nets: Deep Polynomial Neural Networks (2020) https://arxiv.org/abs/2003.03828

      nickpsecurityView on HNParent context

  2. Complexity is a source of income in open source ecosystems (2019) 1 comment

    • The central conceit of the paper (https://arxiv.org/abs/1806.06850) is the "NN<->PR correspondence" from section 6, which argues that since nice nonlinear activation functions like tanh can be arbitrarily well-approximated by polynomials, then the whole stack of convolutional…

      bigbillheckView on HNParent context

  3. Why deep learning works even though it shouldn’t 1 comment

    • That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850

      tbenstView on HNParent context

  4. Differentiable Control Problems 1 comment

    • "The real promise of these methods is to use the universal approximator power of NNs...", still if one is to use a grey-box non-linear model dx/dt = F(x, u, t), why use NNs to characterize F? I would be more comfortable using a polynomial to characterize non-linearity than a…

      pidtunerView on HNParent context

  5. Neural Networks Are Essentially Polynomial Regression 1 comment

    • They revised the paper in response to earlier, uh, response. https://matloff.wordpress.com/2018/07/01/update-on-polynomia... https://arxiv.org/abs/1806.06850

      carapaceView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.