HackerPapers

arXiv:2304.06835cs.DC

Automated Translation and Accelerated Solving of Differential Equations on Multiple GPU Platforms

Utkarsh Utkarsh, Valentin Churavy, Yingbo Ma, Tim Besard, Prakitr Srisuma, Tim Gymnich, Adam R. Gerlach, Alan Edelman, George Barbastathis, Richard D. Braatz, Christopher Rackauckas

Abstract

We demonstrate a high-performance vendor-agnostic method for massively parallel solving of ensembles of ordinary differential equations (ODEs) and stochastic differential equations (SDEs) on GPUs. The method is integrated with a widely used differential equation solver library in a high-level language (Julia's DifferentialEquations.jl) and enables GPU acceleration without requiring code changes by the user. Our approach achieves state-of-the-art performance compared to hand-optimized CUDA-C++ kernels while performing 20--100$\times$ faster than the vectorizing map (vmap) approach implemented in JAX and PyTorch. Performance evaluation on NVIDIA, AMD, Intel, and Apple GPUs demonstrates performance portability and vendor-agnosticism. We show composability with MPI to enable distributed multi-GPU workflows. The implemented solvers are fully featured -- supporting event handling, automatic differentiation, and incorporation of datasets via the GPU's texture memory -- allowing scientists to take advantage of GPU acceleration on all major current architectures without changing their model code and without loss of performance. We distribute the software as an open-source library https://github.com/SciML/DiffEqGPU.jl

arXiv comments
14 figures
Journal reference
Computer Methods in Applied Mechanics and Engineering, Volume 419, 2024
DOI
doi:10.1016/j.cma.2023.116591
Subjects
cs.DCDistributed, Parallel, and Cluster Computing cs.MSMathematical Software math.NANumerical Analysis

On Hacker News

Linked by 3 identifiable commenters in 5 comments across 5 threads, from May 2023 to Aug 2024. Found in 1 HN submission from Dec 2023 to Dec 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Micrograd.jl 1 comment

    • One of the really nice things about Julia for GPU programming is that you can write your own kernels. CUDA.jl isn't just C kernels. This is why (for example) DiffEQGPU.jl is able to be a lot faster than the other GPU based ODE solvers (see https://arxiv.org/abs/2304.06835 for…

      adgjlsfhk1View on HNParent context

  2. Nonlinearsolve.jl: Fast and Robust Solvers for Nonlinear Equations in Julia 1 comment

    • No, there are two types of GPU usage: the ensemble form and the array form. In the array form, you simply make your state variable a GPU and can run all internal operations on the GPU. This is used for GPU acceleration of `f` functions where you have a large state and a…

      ChrisRackauckasView on HNParent context

  3. 2023 was the year that GPUs stood still 1 comment

    • Indeed, and this year we created a system for compiling ODE code not just optimized CUDA kernels but also OneAPI kernels, AMD GPU kernels, and Metal. Peer reviewed version is here (https://www.sciencedirect.com/science/article/abs/pii/S00457...), open access is here…

      ChrisRackauckasView on HNParent context

  4. Translation and accelerated solving of differential equations on GPU platforms 1 comment

    • https://ar5iv.org/abs/2304.06835 Thank you! I guess that's what: > If you are familiar with ar5iv, an arXivLabs collaboration, our HTML offering is essentially bringing this impactful project fully “in-house”. Our ultimate goal is to backfill arXiv’s entire corpus so that every…

      e12eView on HNParent context

  5. 4-Bit Quantization and QLoRA 1 comment

    • We recently showed DiffEqGPU.jl generating customized ODE solver kernels for NVIDIA CUDA, AMD GPUs, Intel OneAPI, and Apple Metal, where for CUDA it matches state of the art (MPGOS) which is about 10x-100x something like Jax/PyTorch (where the performance difference comes from…

      ChrisRackauckasView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.