Predictive Coding Approximates Backprop along Arbitrary Computation Graphs · HackerPapers discussions

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by macrolime

Sun, 12 May 2024 13:24:55 GMT

It's not really all that useful for AI training. The reason why energy is a problem for AI training is that you need a lot of compute at one place, distributing backpropagation doesn't work that well. When the energy problems hit, it's when the datacenter need exceeds the capacity of nearby power plants. If we want an alternative I think we'd need to try alternative training algorithms that are easier to distribute. Predictive coding for example can approximate backprop, but requires more compute. Something in that direction might work https://arxiv.org/abs/2006.04182

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by kelseyfrog

Mon, 11 Sep 2023 15:36:51 GMT

To add, there is a whole field of biologically-plausible learning. Predictive coding is one such method. It is isomorphic to SGD[1], and doesn't require global computation. ie it's backprop-free optimization. Although slower than SGD on silicon, it's massively parallelizible which ends up making it more biologically plausible rather than less. 1. 2020 PREDICTIVE CODING APPROXIMATES BACKPROP ALONG ARBITRARY COMPUTATION GRAPHS https://arxiv.org/pdf/2006.04182.pdf

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by canjobear

Fri, 09 Jun 2023 22:59:42 GMT

Approximately, via predictive coding https://arxiv.org/abs/2006.04182

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — HN submission

Sun, 02 May 2021 14:06:44 GMT

Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs (2020)

65 points; 9 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by etienne618

Thu, 29 Apr 2021 16:28:19 GMT

It seems that backpropagation might be a good abstraction for biological learing rules after all: "local predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules" from https://arxiv.org/abs/2006.04182

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by cs702

Fri, 23 Apr 2021 15:26:06 GMT

> from a longer term perspective Yes. Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training. Recent progress suggests this will happen. See, for example: https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2103.03725 https://arxiv.org/abs/2010.01047 I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense…

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by cs702

Mon, 05 Apr 2021 17:17:49 GMT

EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence, but also for making it trivial to train models in a fully distributed manner, with all learning done locally -- if the results hold. Here's the paper: "Predictive Coding Approximates…

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Approximates Backprop along Arbitrary Computation Graphs — mentioned by canjobear

Sat, 20 Feb 2021 19:41:39 GMT

Backprop isn't biologically plausible but predictive coding is and it approximates backprop. https://arxiv.org/abs/2006.04182

Paper and discussion history. A mention is not an endorsement.