HackerPapers

arXiv:1810.07951cs.PL

Don't Unroll Adjoint: Differentiating SSA-Form Programs

Michael Innes

Abstract

This paper presents reverse-mode algorithmic differentiation (AD) based on source code transformation, in particular of the Static Single Assignment (SSA) form used by modern compilers. The approach can support control flow, nesting, mutation, recursion, data structures, higher-order functions, and other language constructs, and the output is given to an existing compiler to produce highly efficient differentiated code. Our implementation is a new AD tool for the Julia language, called Zygote, which presents high-level dynamic semantics while transparently compiling adjoint code under the hood. We discuss the benefits of this approach to both the usability and performance of AD tools.

arXiv comments
Journal reference
DOI
Subjects
cs.PLProgramming Languages

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from Apr 2019 to Dec 2021. Found in 4 HN submissions from Oct 2018 to Feb 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Trade-Offs in Automatic Differentiation: TensorFlow, PyTorch, Jax, and Julia 1 comment

    • I remember reading an early S4TF manifesto talking about natural language processing, image processing, etc. thinking, that cannot be your audience because that audience already has AD systems which support their domain. Building something that is more general for the sake of…

      ChrisRackauckasView on HNParent context

  2. Useful algorithms that are not optimized by Jax, PyTorch, or TensorFlow 1 comment

    • There is definitely no free lunch, it's good to really delineate the engineering trade-offs you're making! A lot of this work actually comes from the fact that some people I work with were building tools that could efficiently handle dynamic control flow without requiring…

      ChrisRackauckasView on HNParent context

  3. The Accelerating Adoption of Julia 1 comment

    • I am confused why you assume I am a "Julia booster" or use such combative language. I love Python and Jax and use it for much of my research work, I just also like learning of other approaches. Please try to honestly address the sibling comments. We have repeatedly claimed that…

      krastanovView on HNParent context

  4. Demystifying Differentiable Programming (2018) 1 comment

    • I have not read the paper yet but, reading the abstract, the idea of using continuation to store the differentiation information is reminiscent of the technique used in Zygote[0]. Is there some parenty between the ideas ? [0]: https://arxiv.org/abs/1810.07951v4

      nestorDView on HNParent context

  5. A Differentiable Programming System to Bridge ML and Scientific Computing 1 comment

    • JAX is a sophisticated and well implemented high level tracer, combined with a backend compiler that makes use of XLA. High level tracing has some fundamental draw backs in key features we want (efficient scalar AD for example), so the approach being advocated here is doing AD…

      KenoFischerView on HNParent context

  6. Embracing Swift for Deep Learning 1 comment

    • Question for Jeremy: See links below. Did you look at or consider Julia? -- https://julialang.org/blog/2018/12/ml-language-compiler https://arxiv.org/abs/1810.07951

      cs702View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.