How to Start Training: The Effect of Initialization and Architecture — mentioned by SiempreViernes

Thu, 28 Nov 2019 00:12:22 GMT

You're right about the central limit theorem appearing, but series expansions didn't appear; instead it is the fact that the weights are initialized to random values that seems to carry the day. I couldn't find any mention about a trained NN, this is strictly about the initial state. Yang does reference a few papers that supposedly leverage the GP correspondence to gain some insight about how to better initialize a NN, for example this: https://arxiv.org/abs/1803.01719

Paper and discussion history. A mention is not an endorsement.

How to Start Training: The Effect of Initialization and Architecture · HackerPapers discussions

How to Start Training: The Effect of Initialization and Architecture — mentioned by SiempreViernes