Understanding the Difficulty of Training Transformers — mentioned by kmmlng

Thu, 04 Jan 2024 14:53:18 GMT

What do you mean, the development of the "Perception"? Do you mean the Perceptron? In that case, Backprop was invented way later than the Perceptron (see https://people.idsia.ch/~juergen/who-invented-backpropagatio...). I don't see any information in your linked Wikipedia article that supports a bio-inspired origin. In fact, researchers have been wondering whether an equivalent to Backprop might be found in biological brains, but Backprop is widely believed to be biologically implausible (see e.g. https://arxiv.org/pdf/1502.04156.pdf…

Paper and discussion history. A mention is not an endorsement.

Understanding the Difficulty of Training Transformers · HackerPapers discussions

Understanding the Difficulty of Training Transformers — mentioned by kmmlng