HackerPapers

arXiv:2402.06184cs.LG

The boundary of neural network trainability is fractal

Jascha Sohl-Dickstein

Abstract

Some fractals -- for instance those associated with the Mandelbrot and quadratic Julia sets -- are computed by iterating a function, and identifying the boundary between hyperparameters for which the resulting series diverges or remains bounded. Neural network training similarly involves iterating an update function (e.g. repeated steps of gradient descent), can result in convergent or divergent behavior, and can be extremely sensitive to small changes in hyperparameters. Motivated by these similarities, we experimentally examine the boundary between neural network hyperparameters that lead to stable and divergent training. We find that this boundary is fractal over more than ten decades of scale in all tested configurations.

arXiv comments
3 pages, mesmerizing fractals
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing nlin.CDChaotic Dynamics

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 3 threads, from Feb 2024 to Jan 2025. Found in 1 HN submission from Feb 2024 to Feb 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. The Graphics Codex 1 comment

    • > Convolution Nice idea, but I'm also planning a book on advanced cuda programming, and I will cover convolutions there, I'll think if it is interesting to have some kind of cross chapter that is valid for both books. > Noises I cover those in normal mapping > anisotropy I have…

      somethingsomeView on HNParent context

  2. Machine learning dynamics presents fractal patterns 1 comment
  3. Neural network training makes beautiful fractals 2 comments

    • Twitter: https://twitter.com/jaschasd/status/1756930242965606582 ArXiv: https://arxiv.org/abs/2402.06184 Abstract: "Some fractals -- for instance those associated with the Mandelbrot and quadratic Julia sets -- are computed by iterating a function, and identifying the boundary…

      telotortiumView on HNParent context

    • The sister paper[1] contains the fractal dimensions of some of the images using a boxcount paper. [1]: https://arxiv.org/pdf/2402.06184.pdf

      bluetintedfortView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.