HackerPapers

arXiv:2304.12210cs.LG

A Cookbook of Self-Supervised Learning

Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, Micah Goldblum

Abstract

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Sep 2023 to Jun 2024. Found in 2 HN submissions from Apr 2023 to Apr 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Terence Tao on proof checkers and AI programs 1 comment

    • > but the current bottleneck is figuring out how to get a model to train itself against its own output without human intervention this is sort of humorous, because the entire current generation of Transformer-based encoders was built specifically to get "a model to train itself…

      mistrial9View on HNParent context

  2. Self-supervised learning: The dark matter of intelligence (2021) 1 comment

    • This article was written in 2021 when masked language models have been successfully applied in nlp(Bert, word2vec, glove, etc). However at the time, it was unclear how the same technique could be applied to vision tasks because unlike language which has a limited vocab, you…

      isaacfungView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.