Variable Computation in Recurrent Neural Networks · HackerPapers discussions

Variable Computation in Recurrent Neural Networks — mentioned by thesz

Mon, 22 Sep 2025 21:20:04 GMT

> This paper addresses the challenge by asking: how can we trade off more compute for less data? Autoregressive models are not matched by compute and this is the major drawback. There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study. [1] https://arxiv.org/abs/1611.06188 They fixed number of time steps instead of varying it, and got better results. Unfortunately, I forgot the title of that ablation paper.

Paper and discussion history. A mention is not an endorsement.

Variable Computation in Recurrent Neural Networks — mentioned by thesz

Wed, 30 Mar 2022 06:44:30 GMT

Here it is: https://arxiv.org/abs/1611.06188 RNN outputs "confidence" bit which can guide computation to perform more steps to obtain more confidence in the result. Essentially, RNN asks "let me think about that some more". But, separate ablation study found that if you just drop confidence bit altogether and allow RNN to compute some more every time (e.g., always perform 4 computations on single input for 1 output), you get same or better results without extra complexity of training. There is also Microsoft Research's paper I can't find right now about the variable computation for image…

Paper and discussion history. A mention is not an endorsement.