HackerPapers

arXiv:2212.13345cs.LG

The Forward-Forward Algorithm: Some Preliminary Investigations

Geoffrey Hinton

Abstract

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities. If the positive and negative passes could be separated in time, the negative passes could be done offline, which would make the learning much simpler in the positive pass and allow video to be pipelined through the network without ever storing activities or stopping to propagate derivatives.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from Jan 2023 to Jan 2025. Found in 2 HN submissions from Jan 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Antiqua et Nova: Note on the relationship between AI and human intelligence 1 comment

    • While there are some similarities, yes, our brains definitely don't have anything akin to backpropagation, which is the critical mechanism for how current AI models learn. Hinton has some research on a forward-forward learning paradigm [1], which might be closer to how our…

      ewheelerView on HNParent context

  2. Training of Physical Neural Networks 1 comment

    • This is “Mortal Computation” coined in Hinton’s The Forward-Forward Algorithm: Some Preliminary Investigations https://arxiv.org/abs/2212.13345.

      robertsdionneView on HNParent context

  3. Storm: LLM system that researches a topic and generates full-length wiki article 1 comment

    • Here's what I envision (note: impossible with current state of the art) A model that can be incrementally trained (this is the bit we're missing) hosted by a nonprofit, belonging to "we the people" (like wikipedia). The training process could be done a little like wikipedia talk…

      pksebbenView on HNParent context

  4. Geoffrey Hinton: “We're toast. This is the actual end of history” 1 comment

    • > In May 2023, a colonel in the US Air Force revealed a simulation in which an AI-enabled drone, tasked with attacking surface-to-air missiles but acting under the authority of a human operator, attacked the operator instead. (The US Air Force later denied the story.) This casts…

      pksebbenView on HNParent context

  5. MeZO: Fine-Tuning Language Models with Just Forward Passes 1 comment

    • sounds cool, i wonder if its possible to do similar with "The Forward-Forward Algorithm" by Geoff Hilton maybe on a few add on layers like with lora [0]: https://arxiv.org/pdf/2212.13345.pdf

      born-jreView on HNParent context

  6. Geoffrey Hinton publishes new deep learning algorithm 1 comment

    • The article links to an old draft of the paper (it seems that the results in 4.1 couldn't be replicated). The arxiv has a more recent one: https://arxiv.org/abs/2212.13345

      cschmidView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.