ResNet strikes back: An improved training procedure in timm · HackerPapers discussions

ResNet strikes back: An improved training procedure in timm — mentioned by godelski

Tue, 06 Aug 2024 08:26:48 GMT

> The rule in ML There's definitely attempts to revive things (in the general sense, not just GANs), but most successes appear to come from large labs dedicating equal computational resources to the older models and often by changing names. This can make things more confusing and make things appear to be changing faster, but once you can see this, you'll have an easier time keeping up (so being new, watch out for this). I'll give some examples that are easier to read[0-2] (i.e. don't need expert knowledge to understand the nuances). As an insider (ML researcher), my complaint isn't so much…

Paper and discussion history. A mention is not an endorsement.

ResNet strikes back: An improved training procedure in timm — mentioned by godelski

Wed, 29 Nov 2023 19:35:50 GMT

> Also, we know that transformers can scale Do we have strong evidence that other models don't scale or have we just put more time into transformers? Convolutional resnets look to scale on vision and language: (cv) https://arxiv.org/abs/2301.00808, (cv) https://arxiv.org/abs/2110.00476, (nlp) https://github.com/HazyResearch/safari MLPs also seem to scale: (cv) https://arxiv.org/abs/2105.01601, (cv) https://arxiv.org/abs/2105.03404 I mean I don't see a strong reason to turn away from attention as well but I also don't think anyone's thrown a billion parameter MLP or Conv model at a problem.…

Paper and discussion history. A mention is not an endorsement.

ResNet strikes back: An improved training procedure in timm — mentioned by godelski

Mon, 10 Apr 2023 00:02:20 GMT

This is still being pursued. Ross Wightmann's timm[0,1] package (now on Hugging Face) has done a lot of this. There's also a V2 of ConvNext[2]. Ross does write about this a lot on Twitter fwiw. I should also mention that there are still many transformer based networks that still beat convs. So there probably won't be a resurgence in convs until someone can show that there's a really strong reason for them. They have some advantages but they also might not be flexible enough for the long range tasks in segmentation and detection. But maybe they are. FAIR definitely did great work with…

Paper and discussion history. A mention is not an endorsement.

ResNet strikes back: An improved training procedure in timm — mentioned by mrintellectual

Wed, 13 Apr 2022 18:33:22 GMT

> ViT models are outperforming CNNs in terms of computational efficiency and accuracy, achieving highly competitive performance in tasks like image classification, object detection, and semantic image segmentation. Since then, this has been show to be untrue. Using more modern training techniques along with depthwise convolutions (https://arxiv.org/abs/2201.03545) results in equal if not better performance on vision tasks. Improved training methodologies have also been shown to boost the accuracy of ResNet50 - an 6-year-old pure convolutional architecture - on ImageNet-1k by over 5%…

Paper and discussion history. A mention is not an endorsement.

ResNet strikes back: An improved training procedure in timm — HN submission

Mon, 04 Oct 2021 09:02:34 GMT

ResNet strikes back: An improved training procedure in timm

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.