Don't Unroll Adjoint: Differentiating SSA-Form Programs · HackerPapers discussions

Don't Unroll Adjoint: Differentiating SSA-Form Programs — mentioned by ChrisRackauckas

Sat, 25 Dec 2021 16:30:10 GMT

I remember reading an early S4TF manifesto talking about natural language processing, image processing, etc. thinking, that cannot be your audience because that audience already has AD systems which support their domain. Building something that is more general for the sake of being more general is never a good idea, that's bad engineering. It had a lot of great ideas, and indeed the dev experience seemed nice. But I would venture to guess that the Google overlords had to question what the true value of S4TF in that light. "Standard ML" cannot be your target audience if you want to work on AD…

Paper and discussion history. A mention is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — mentioned by ChrisRackauckas

Thu, 22 Jul 2021 17:18:43 GMT

There is definitely no free lunch, it's good to really delineate the engineering trade-offs you're making! A lot of this work actually comes from the fact that some people I work with were building tools that could efficiently handle dynamic control flow without requiring tracing (see the description of Zygote.jl https://arxiv.org/abs/1810.07951). I had to bring up the question: why? It's much harder to build, needs more machinery, and in some cases can make less assumptions/less fusions (a general form of vmap is much harder for example if you cannot trace, see KernelAbstractions.jl for…

Paper and discussion history. A mention is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — mentioned by krastanov

Wed, 21 Oct 2020 14:48:07 GMT

I am confused why you assume I am a "Julia booster" or use such combative language. I love Python and Jax and use it for much of my research work, I just also like learning of other approaches. Please try to honestly address the sibling comments. We have repeatedly claimed that tools like Zygote.jl can autodifferentiate efficiently things that Jax can not (without a lot of extra special code and hand-defined backprop methods), e.g., an array of structs with scalar and vector properties over which a scalar cost is defined. Just give examples, so that we can both learn something new about these…

Paper and discussion history. A mention is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — mentioned by nestorD

Mon, 17 Feb 2020 07:51:22 GMT

I have not read the paper yet but, reading the abstract, the idea of using continuation to store the differentiation information is reminiscent of the technique used in Zygote[0]. Is there some parenty between the ideas ? [0]: https://arxiv.org/abs/1810.07951v4

Paper and discussion history. A mention is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — mentioned by KenoFischer

Fri, 19 Jul 2019 20:23:26 GMT

JAX is a sophisticated and well implemented high level tracer, combined with a backend compiler that makes use of XLA. High level tracing has some fundamental draw backs in key features we want (efficient scalar AD for example), so the approach being advocated here is doing AD as a compiler transform on the original source language. Mike has some details in an earlier paper: https://arxiv.org/abs/1810.07951. As a shameless plug, you can plug (heh) Julia into XLA as well, e.g. for targeting TPUs (https://arxiv.org/abs/1810.09868). In fact, Julia and JAX are probably better at that than…

Paper and discussion history. A mention is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — mentioned by cs702

Mon, 22 Apr 2019 00:50:30 GMT

Question for Jeremy: See links below. Did you look at or consider Julia? -- https://julialang.org/blog/2018/12/ml-language-compiler https://arxiv.org/abs/1810.07951

Paper and discussion history. A mention is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — HN submission

Fri, 08 Feb 2019 23:17:45 GMT

Differentiating SSA-Form Programs in Julia (2018) [pdf]

87 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — HN submission

Wed, 24 Oct 2018 13:39:55 GMT

Don't Unroll Adjoint: Differentiating SSA-Form Programs

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — HN submission

Tue, 23 Oct 2018 11:59:43 GMT

Don't Unroll Adjoint: Differentiating SSA-Form Programs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Don't Unroll Adjoint: Differentiating SSA-Form Programs — HN submission

Sat, 20 Oct 2018 01:34:13 GMT

Don't Unroll Adjoint: Differentiating SSA-Form Programs

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.