Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning · HackerPapers discussions

Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning — mentioned by orange3xchicken

Fri, 06 Aug 2021 15:34:27 GMT

This is basically adversarial training, which is a typical (& very practical) benchmark heuristic defense for this problem. An ongoing question is to precisely characterize when and how AT works. The line of work has also proved to be very fruitful for the theoretical community & has produced very general results about problems which can be solved by neural networks, but not other techniques- e.g. kernel methods. https://arxiv.org/abs/2001.04413

Paper and discussion history. A mention is not an endorsement.

Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning — mentioned by orange3xchicken

Tue, 09 Feb 2021 04:25:23 GMT

To add on: Zeyuan Allen-Zhu and Yuanzhi Li had a great theoretical paper earlier last year proving a fundamental discrepancy between the concept classes learnable by kernel methods and deep learning models. https://arxiv.org/abs/2001.04413

Paper and discussion history. A mention is not an endorsement.

Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning — mentioned by blackbear_

Mon, 07 Dec 2020 09:18:57 GMT

Interesting to compare this with [1], which shows that for some concept classes neural networks trained with SGD are much easier to train (i.e., require less data). In other words, even though these two types of model are in a way equivalent, one can be much easier to train than the other for certain concepts (no free lunch). [1] https://arxiv.org/abs/2001.04413

Paper and discussion history. A mention is not an endorsement.