Neural Networks and the Chomsky Hierarchy · HackerPapers discussions

Neural Networks and the Chomsky Hierarchy — mentioned by cma

Sun, 12 Jul 2026 18:09:49 GMT

This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better: Neural Networks and the Chomsky Hierarchy https://arxiv.org/abs/2207.02098

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by YeGoblynQueenne

Thu, 14 Nov 2024 14:26:51 GMT

Thanks, that's what I meant. a^nb^n is a standard test of learnability. That stuff is still absolutely relevant, btw. Some DL people like to dismiss it as irrelevant but that's just because they lack the background to appreciate why it matters. Also: the arrogance of youth (hey I've already been a postdoc for a year, I'm ancient). Here's a recent paper on Neural Networks and the Chomsky Hierarchy that tests RNNs and Transformers on formal languages (I think it doesn't test on a^nb^n directly but tests similar a-b based CF languages): https://arxiv.org/abs/2207.02098 And btw that's a good…

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by wslh

Tue, 17 Sep 2024 12:30:05 GMT

My 2 cents: Since LLMs (Large Language Models) operate as at least a subset of Turing machines (which recognize recursively enumerable languages), the chain of thought (CoT) approach could be equivalent to or even more expressive than that subset. In fact, CoT could perfectly be a Turing machine. If we leave CoT aside for a moment, it's worth exploring the work discussed in the paper "Neural Networks and the Chomsky Hierarchy"[1], which analyzes how neural networks (including LLMs) map onto different levels of the Chomsky hierarchy, with a particular focus on their ability to recognize formal…

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by cma

Wed, 10 Apr 2024 12:44:36 GMT

Deepmind found transformers get beat out on learning to reproduce outs towards the general computer end of the Chomsky formal language heirarchy by neural turing machines and even LSTMs: https://arxiv.org/abs/2207.02098

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by cma

Sun, 07 Apr 2024 13:51:56 GMT

Deepmind did a study of which language classes different neural network architectures can learn with generalization: https://arxiv.org/abs/2207.02098 ("Neural Networks and the Chomsky Hierarchy") LSTMs could learn slightly deeper into the Chomsky hierarchy than transformers, neural turing machines could learn them all including recursively enumerable (context free grammar or turing machine to produce). Transformers did outperform LSTMs in something in an alternate hierarchy if I remember. It's not conclusive since it is not training indefinitely, there are formal proofs that many…

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by cma

Tue, 13 Jun 2023 18:59:30 GMT

Deep Mind had this paper on that, showing weaknesses of transformers relative to other models in generalizing from example productions from different tiers of the hierarchy: https://arxiv.org/abs/2207.02098

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by jabowery

Wed, 07 Jun 2023 18:32:18 GMT

The point of this "IQ Test" is to set a relatively low-bar for passing the IQ test question so that even intellectually lazy people can get an intuitive feel for the limitation of Transformer models. This limitation has been pointed out formally by the DeepMind paper "Neural Networks and the Chomsky Hierarchy". https://arxiv.org/abs/2207.02098 The general principle may be understood in terms of the approximation of Solomonoff Induction by natural intelligence during the activity known as "data driven science" aka "The Unreasonable Effectiveness of Mathematics In the Natural Sciences".…

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by cma

Sat, 13 May 2023 16:14:52 GMT

There are also limits to the sequence to sequence things they can effectively learn, in some cases weaker than LSTMs: https://arxiv.org/abs/2207.02098

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by intalentive

Wed, 26 Apr 2023 06:12:15 GMT

They can't even learn parity checks. https://arxiv.org/abs/2207.02098

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by shawntan

Tue, 18 Apr 2023 05:42:56 GMT

Recurrency will help, see the empirical results in this paper: https://arxiv.org/abs/2207.02098. LSTMs can solve the PARITY problem, generalising to longer lengths of the problem without issue. The issue with re-introducing recurrency is that no one wants to bear the O(N) growth in computation time. I wonder if we are settling in for a local-optima of bounded-depth architecture for a while due to the obsession with rapid scaling.

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by cma

Wed, 15 Feb 2023 03:52:58 GMT

Some big limitations of transformers as well: Neural Networks and the Chomsky Hierarchy [Deep Mind 2022] https://arxiv.org/abs/2207.02098

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — HN submission

Mon, 23 Jan 2023 04:55:38 GMT

Neural Networks and the Chomsky Hierarchy

164 points; 79 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Neural Networks and the Chomsky Hierarchy — mentioned by emiliano2022

Sat, 20 Aug 2022 18:51:07 GMT

This is a very interesting article, in particular because it brings up that whole Chomsky topic of the grammar hierarchy (see https://arxiv.org/pdf/2207.02098.pdf). I've been intuiting that linguistics are at the root of AGI for sometime now, but there is a very unproductive resistance to Chomsky's ideas in the various communities that have topic overlap. It really seems apparent, to me, that symbols (a universal grammar?) are needed in order to learn & adapt to the environment. If true, then the question becomes how to evolve that universality from nothing.

Paper and discussion history. A mention is not an endorsement.

Neural Networks and the Chomsky Hierarchy — HN submission

Sat, 16 Jul 2022 19:25:01 GMT

Neural Networks and the Chomsky Hierarchy by DeepMind

5 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.