The Unreasonable Ineffectiveness of the Deeper Layers · HackerPapers discussions

The Unreasonable Ineffectiveness of the Deeper Layers — mentioned by sigmoid10

Tue, 09 Jul 2024 15:22:47 GMT

Other research from Meta FAIR actually suggests that you should prune deeper layers if you want to improve performance while maintaining accuracy [1]. So there must be a cutoff point for smaller networks where this approach still works, otherwise the results are contradictory. Or we could drastically improve these new models even further. [1] https://arxiv.org/html/2403.17887v1

Paper and discussion history. A mention is not an endorsement.

The Unreasonable Ineffectiveness of the Deeper Layers — HN submission

Tue, 02 Apr 2024 07:52:30 GMT

The Unreasonable Ineffectiveness of the Deeper Layers

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Unreasonable Ineffectiveness of the Deeper Layers — mentioned by imtringued

Sat, 30 Mar 2024 10:36:26 GMT

You can just delete whole layers, because the point of residual layers is to make the model learn the hyperparameter "layer count" automatically. Compare this to the absence of residual layers where the model must use all layers. Then you will have to get the layer count perfect, but this isn't possible, since each data point might benefit from a different layer count. The extra layers therefore exist primarily so the model becomes robust to poorly chosen hyper parameters. You still need a minimum amount of layers, but that isn't the problem. https://arxiv.org/abs/2403.17887

Paper and discussion history. A mention is not an endorsement.

The Unreasonable Ineffectiveness of the Deeper Layers — mentioned by underlines

Wed, 27 Mar 2024 22:09:41 GMT

This paper partially finds disagreeing evidence: https://arxiv.org/abs/2403.17887

Paper and discussion history. A mention is not an endorsement.

The Unreasonable Ineffectiveness of the Deeper Layers — HN submission

Wed, 27 Mar 2024 11:33:45 GMT

The Unreasonable Ineffectiveness of the Deeper Layers

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.