Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks — mentioned by shawntan

Tue, 14 Oct 2025 02:00:50 GMT

This would not help if no proper constraints are established on what data can and cannot be trained on. And maybe just figuring out what the goal of the benchmark is. If it is to test generalisation capability, then what data the model being evaluated is trained on is crucial to making any conclusions. Look at the construction of this synthetic dataset for example: https://arxiv.org/pdf/1711.00350

Paper and discussion history. A mention is not an endorsement.

Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks · HackerPapers discussions

Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks — mentioned by shawntan