Searching for Activation Functions · HackerPapers discussions

Searching for Activation Functions — mentioned by f_devd

Wed, 08 Feb 2023 19:21:12 GMT

Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns out that doesn't magically make make neural networks orders better. [0]: https://en.m.wikipedia.org/wiki/Swish_function [1]: https://arxiv.org/abs/1710.05941

Paper and discussion history. A mention is not an endorsement.

Searching for Activation Functions — mentioned by Hawkenfall

Fri, 15 Nov 2019 15:59:56 GMT

A more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941

Paper and discussion history. A mention is not an endorsement.

Searching for Activation Functions — HN submission

Thu, 02 Nov 2017 13:22:47 GMT

The “Swish” Activation Function: Like ReLU but Better

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Searching for Activation Functions — HN submission

Mon, 30 Oct 2017 17:40:26 GMT

Swish: A Self-Gated Activation Function

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Searching for Activation Functions — HN submission

Fri, 27 Oct 2017 09:24:24 GMT

Swish: A Self-Gated Activation Function

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Searching for Activation Functions — HN submission

Wed, 18 Oct 2017 10:19:53 GMT

Swish: A Self-Gated Activation Function

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.