Deep Symbolic Regression for Recurrent Sequences · HackerPapers discussions

Deep Symbolic Regression for Recurrent Sequences — mentioned by Veedrac

Tue, 29 Mar 2022 19:46:40 GMT

Nothing prevents it, no. Transformers are certainly capable of learning mathematical tasks; consider [1] as an example, which uses big but regular token lengths. Alternatively you could just scale 'till the problem solves itself. [1] https://arxiv.org/abs/2201.04600

Paper and discussion history. A mention is not an endorsement.

Deep Symbolic Regression for Recurrent Sequences — mentioned by Isinlor

Fri, 11 Feb 2022 10:33:08 GMT

Sure: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Both of these models can generalize to numbers it have not seen.

Paper and discussion history. A mention is not an endorsement.

Deep Symbolic Regression for Recurrent Sequences — mentioned by Isinlor

Fri, 11 Feb 2022 10:31:29 GMT

Take a look at this paper: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 If you look at embedding visualization it is very clear that the model learns order of numbers. (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) There is also: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Again, looking at visualizations the model very clearly grasps the structure of the function it models.

Paper and discussion history. A mention is not an endorsement.

Deep Symbolic Regression for Recurrent Sequences — mentioned by ShamelessC

Sun, 30 Jan 2022 21:15:44 GMT

tl;dr - They model operations (e.g. multiply, add, cosine, sqrt) as tokens, and use a transformer to predict a recursive function from (incomplete) integer sequences (and float sequences, separately). This is not dissimilar from e.g. DALL-E where they model images as NLP-style tokens and use a transformer to predict image tokens (operations, respectively) from text tokens (list of ints/floats, respectively). This demo lets you interact with the model. Pretty cool stuff. Refreshing to see symbolic problems being tackled with deep learning. Yannic Kilcher video with one of the authors…

Paper and discussion history. A mention is not an endorsement.