A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions · HackerPapers discussions

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions — mentioned by ChrisRackauckas

Sun, 26 Jun 2022 18:12:19 GMT

Sundials does have forward sensitivities, though the implementation does not SIMD the calculations of the primal with the dual parts like direct AD of the solver does. This makes it a bit worse on forward mode than direct differentiation of a solver (with an optimizing compiler) would do, which was something that took us a few years to really pin down (we had to implement our own to really benchmark it, and got the same phenomena demonstrated in https://arxiv.org/abs/1812.01892). The nice thing about the direct forward mode is that it is easier to opt the extra equations out of the linear…

Paper and discussion history. A mention is not an endorsement.

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions — mentioned by ChrisRackauckas

Fri, 22 Oct 2021 10:37:14 GMT

Yes, that's mostly right (instead of PETSc put FATODE since PETSc TS Adjoint was only published in 2019, but it's based heavily on FATODE's techniques https://epubs.siam.org/doi/10.1137/130912335?mobileUi=0). And that's why the Julia tools were so ready for it: we already had adjoint sensitivity analysis (implemented for parameter estimation in systems pharmacology), so neural ODEs were a freebee. Similar to DEQs using the adjoint of a nonlinear solve: that means a DEQ is just a neural network inside of a Julia nonlinear solver and you call the gradients and you'll hit the right overloads…

Paper and discussion history. A mention is not an endorsement.

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions — mentioned by ChrisRackauckas

Tue, 18 Dec 2018 02:18:47 GMT

We are building tooling which will allow you to put together differential equations and neural networks in any way you please. This is just one way of putting the two together. The applications we were looking at are different, but it utilize similar tooling. We have forward-mode, reverse-mode, forward sensitivity analysis, and adjoint sensitivity analysis (the method mentioned here, which is actually from at least the 90's) implemented and tested with the DifferentialEquations.jl integrators. We had a recent paper exploring the timing differences between the different sensitivity analysis…

Paper and discussion history. A mention is not an endorsement.

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions — mentioned by ChrisRackauckas

Sun, 16 Dec 2018 00:16:34 GMT

Yes, we've had this working in Julia for a few years now (with Flux.jl and ReverseDiff.jl tracing-based reverse-mode) and it's surprising that wasn't know because it's been pretty widely publicized (and mentioned to the authors). In fact, we even have a preprint showing that reverse-mode adjoint isn't what you want to use in most scenarios: https://arxiv.org/abs/1812.01892 We identified that tracing-based reverse-mode adjoint + solving without mutation is not competitive with the other sensitivity analysis methods (this is the method which they have implemented in this repository) when…

Paper and discussion history. A mention is not an endorsement.

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions — mentioned by ChrisRackauckas

Thu, 06 Dec 2018 01:31:04 GMT

This is it: https://arxiv.org/abs/1812.01892

Paper and discussion history. A mention is not an endorsement.