HackerPapers

arXiv:1801.10277cs.DC

Cataloging the Visible Universe through Bayesian Inference at Petascale

Jeffrey Regier, Kiran Pamnany, Keno Fischer, Andreas Noack, Maximilian Lam, Jarrett Revels, Steve Howard, Ryan Giordano, David Schlegel, Jon McAuliffe, Rollin Thomas, Prabhat

Abstract

Astronomical catalogs derived from wide-field imaging surveys are an important tool for understanding the Universe. We construct an astronomical catalog from 55 TB of imaging data using Celeste, a Bayesian variational inference code written entirely in the high-productivity programming language Julia. Using over 1.3 million threads on 650,000 Intel Xeon Phi cores of the Cori Phase II supercomputer, Celeste achieves a peak rate of 1.54 DP PFLOP/s. Celeste is able to jointly optimize parameters for 188M stars and galaxies, loading and processing 178 TB across 8192 nodes in 14.6 minutes. To achieve this, Celeste exploits parallelism at multiple levels (cluster, node, and thread) and accelerates I/O through Cori's Burst Buffer. Julia's native performance enables Celeste to employ high-level constructs without resorting to hand-written or generated low-level code (C/C++/Fortran), and yet achieve petascale performance.

arXiv comments
accepted to IPDPS 2018
Journal reference
DOI
Subjects
cs.DCDistributed, Parallel, and Cluster Computing astro-ph.IMInstrumentation and Methods for Astrophysics

On Hacker News

Linked by 2 identifiable commenters in 3 comments across 3 threads, from Oct 2018 to Feb 2021. Found in 1 HN submission from Sep 2020 to Sep 2020. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. New 10 Terapixel Image of the Night Sky Contains 1B Galaxies 1 comment

    • Counting the number of galaxies in these image datasets is actually a highly non-trivial, supercomputer-scale problem. I worked on this a few years back! Paper link for those interested: https://arxiv.org/pdf/1801.10277.pdf

      KenoFischerView on HNParent context

  2. Why Julia 1 comment

    • And these were not published on their website: https://arxiv.org/pdf/1801.10277.pdf http://proceedings.mlr.press/v37/regier15.pdf https://arxiv.org/pdf/1803.00113.pdf I had to go so far as to read the project Github page to find them.

      ethelwardView on HNParent context

  3. Parallel Supercomputing for Astronomy with Julia 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.