HackerPapers

arXiv:1806.07366cs.LG

Neural Ordinary Differential Equations

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud

Abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

arXiv comments
v6: Corrected Fig. 3c function-evaluation counts (thanks to Vanni et al.); typo fixes
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 11 identifiable commenters in 11 comments across 11 threads, from Jun 2018 to Mar 2024. Found in 2 HN submissions from Dec 2018 to Dec 2018. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments11 threads

  1. On the Double-Slit Experiment and Quantum Interference in the Wolfram Model (2020) 1 comment

    • This paper [1] models some simple (r) NN as ODEs, and uses ODE tools to train and for inference. It’s a start. [1] https://arxiv.org/abs/1806.07366

      rfonsecaView on HNParent context

  2. Where is Noether's principle in machine learning? 1 comment

    • People have mentioned the discrete - continuous tradeoff. One way to bridge that gap would be to use https://arxiv.org/abs/1806.07366 - they draw an equivalence between vanilla (FC layer) neural nets of constant width with differential equations, and then use a differential…

      Scene_Cast2View on HNParent context

  3. Why train when you can optimize? 1 comment
  4. Retrospective review of Gödel, Escher, Bach (1996) [pdf] 1 comment

    • Yeah. This kind of mirrors my thought that our visual cortex is excellent at split-second image recognition, for things like identifying predators, dangers, etc, and this lack of "thinking" is why our current generation of neural networks has matched the performance of humans on…

      rubatugaView on HNParent context

  5. Deep Learning for Symbolic Mathematics 1 comment

    • Neural ODEs? "Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. " - https://arxiv.org/abs/1806.07366

      angel_jView on HNParent context

  6. ‘A Swiss cheese-like material’ that can solve equations 1 comment

    • This work focuses on steady-state computing, but it could also be interesting to use transient physical behaviors to process time-varying signals Maybe by modelling dynamical systems as "neural nets" as in: https://arxiv.org/abs/1806.07366 and https://arxiv.org/abs/1808.08412 Or…

      damipView on HNParent context

  7. Deep Learning State of the Art (2019) [video] 1 comment

    • Nice work. I can think of only two things that are missing: * Normalizing flows - e.g., https://arxiv.org/abs/1605.08803 , https://arxiv.org/abs/1807.03039 , among many others * ODEnets and continous normalizing flows - https://arxiv.org/abs/1806.07366

      cs702View on HNParent context

  8. Tensor Considered Harmful Part 2 1 comment

    • >We've got a winner. Tensors "do" something; they're functions. And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the…

      joshuamortonView on HNParent context

  9. A new neural network design could overcome challenges in AI 1 comment

    • The paper the article is based on is here: https://arxiv.org/abs/1806.07366 And there was mention on HN about a month ago of a PyTorch implementation for ODEs (the name for this new NN design) here: https://github.com/rtqichen/torchdiffeq/blob/master/README.m...

      nmstokerView on HNParent context

  10. Building a Language and Compiler for Machine Learning 1 comment

    • Neural ODE solver just announced as a Best Paper at NeurIPS 2018 conference. What's interesting is that not only do preliminary results yield significantly less error in prediction vs RNNs. Differentiable models allow for extrapolations well beyond range of observable test data…

      ArtWombView on HNParent context

  11. Polynomial Regression as an Alternative to Neural Nets 1 comment

    • I've long wondered about this - I see and understand the comments by people pointing out this paper may a bit shallow - in the same sort of vein, I have this on my desk to grind thru - interested if anyone else has looked at it -- https://arxiv.org/abs/1806.07366

      MarkMMullinView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.