Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks · HackerPapers discussions

Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks — mentioned by cs702

Fri, 23 Apr 2021 15:26:06 GMT

> from a longer term perspective Yes. Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training. Recent progress suggests this will happen. See, for example: https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2103.03725 https://arxiv.org/abs/2010.01047 I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense…

Paper and discussion history. A mention is not an endorsement.

Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks — mentioned by babel_

Mon, 05 Apr 2021 22:51:39 GMT

Interesting follow up reading: "Relaxing the Constraints on Predictive Coding Models" (https://arxiv.org/abs/2010.01047), from the same authors. Looks at ways to remove neurological implausibility from PCM and achieve comparable results. Sadly they only do MNIST in this one, and are not as ambitious in testing on multiple architectures and problems/datasets, but the results are still very interesting and it covers some of the important theoretical and biological concerns. "Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks"…

Paper and discussion history. A mention is not an endorsement.