Weight Agnostic Neural Networks · HackerPapers discussions

Weight Agnostic Neural Networks — mentioned by bob1029

Sat, 08 Nov 2025 17:11:03 GMT

> While playing around with this setup, I tried re-training the network with the activation function for the first layer replaced with sin(x) and it ends up working pretty much the same way. There is some evidence that the activation functions and weights can be arbitrarily selected assuming you have a way to evolve the topology of the network. https://arxiv.org/abs/1906.04358

Paper and discussion history. A mention is not an endorsement.

Weight Agnostic Neural Networks — mentioned by myownpetard

Fri, 07 Oct 2022 20:48:36 GMT

There is a great paper, Weight Agnostic Neural Networks [0], that explores this topic. They experiment with using a single shared weight for a network while using an evolutionary algorithm to find architectures that are themselves biased towards being effective on specific problems. The upshot is that once you've found an architecture that is already biased towards solving a specific problem, then the training of the weights is faster and results in better performance. From the abstract, "...In this work, we question to what extent neural network architectures alone, without learning any…

Paper and discussion history. A mention is not an endorsement.

Weight Agnostic Neural Networks — mentioned by puttycat

Fri, 07 Oct 2022 17:08:05 GMT

Good overview. At the other extreme, some recent works [1,2] show why it’s sometime better to scale down instead of up, especially for some humanlike capabilities like generalization: [1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00489... [2] https://arxiv.org/abs/1906.04358

Paper and discussion history. A mention is not an endorsement.

Weight Agnostic Neural Networks — mentioned by xyproto

Tue, 25 Jan 2022 22:21:48 GMT

How are Hypernetworks different from weight agnostic neural networks? WANN paper: https://arxiv.org/pdf/1906.04358.pdf

Paper and discussion history. A mention is not an endorsement.