HackerPapers

arXiv:1912.03263cs.LG

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, Kevin Swersky

Abstract

We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may beused and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, andout-of-distribution detection while also enabling our models to generate samplesrivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and presentan approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-artin both generative and discriminative learning within one hybrid model.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 7 comments across 7 threads, from Sep 2020 to Feb 2025.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Softmax forever, or why I like softmax 1 comment

    • For people interested in the softmax, log sum exp and energy models, have a look at "Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One" [1] [1]: https://arxiv.org/abs/1912.03263

      mauritsView on HNParent context

  2. Fun times with energy-based models 1 comment

    • EBMs show up all over the place, apparently even your classifier is an EBM :) (https://arxiv.org/abs/1912.03263).

      mpmiskoView on HNParent context

  3. A new type of neural network is more interpretable 1 comment

    • Thanks. I am passionate haha. I do like to educate, but it is often difficult to do on these forms as I do not know peoples backgrounds. If you throw too high level at them, I find it does more harm than good. I also find that while I know many here took things like calculus…

      godelskiView on HNParent context

  4. The Path to StyleGan2 – Implementing the Progressive Growing GAN 1 comment

    • > The rule in ML There's definitely attempts to revive things (in the general sense, not just GANs), but most successes appear to come from large labs dedicating equal computational resources to the older models and often by changing names. This can make things more confusing…

      godelskiView on HNParent context

  5. Understanding deep learning requires rethinking generalization 1 comment

    • But BERT isn't a generative model? Unless you mean it's trained similar to a generative model by predicting the missing tokens? Also an interesting paper about hybrid energy based models: https://arxiv.org/pdf/1912.03263.pdf It's possible to have a model be both a classifier +…

      rubatugaView on HNParent context

  6. Show HN: Build a DIY license plate reader with machine learning 1 comment

    • For neural networks unless it's calibrated otherwise the confidence score has little meaning to the accuracy. [1] see figure 4 and you will understand what I mean [1] https://arxiv.org/pdf/1912.03263.pdf

      blackcat201View on HNParent context

  7. Score matching with Langevin Sampling: a new contender to GANs 1 comment

    • Reminds me of JEMs from Grathwohl et al last year [0]. They train a generative model by treating it an an Energy Based Model using stochastic langevin gradients. I’m curious as to how it relates to the models in this post. [0] https://arxiv.org/abs/1912.03263

      rjeliView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.