Universal pre-training by iterated random computation · HackerPapers discussions

Universal pre-training by iterated random computation — mentioned by benob

Thu, 19 Mar 2026 14:19:35 GMT

Reminds me of "Universal pre-training by iterated random computation" https://arxiv.org/pdf/2506.20057, with bit less formal approach. I wonder if there is a closed-form solution for those kinds of initialization methods (call them pre-training if you wish). A solution that would allow attention heads to detect a variety of diverse patterns, yet more structured than random init.

Paper and discussion history. A mention is not an endorsement.

Universal pre-training by iterated random computation — mentioned by ACCount37

Wed, 08 Oct 2025 01:09:28 GMT

The brain implements some kind of fairly general learning algorithm, clearly. There's too little data in the DNA to wire up 90 billion neurons the way we can just paste 90 billion weights into a GPU over a fiber optic strand. But there's a lot of innate scaffolding that actually makes the brain learn the way it does. Things like bouba and kiki, instincts, all the innate little quirks and biases - they add up to something very important. For example, we know from neuroscience that humans implement something not unlike curriculum learning - and a more elaborate version of it than what we use…

Paper and discussion history. A mention is not an endorsement.

Universal pre-training by iterated random computation — mentioned by benob

Thu, 04 Sep 2025 07:57:59 GMT

Stop thinking about text being the data. There are so many other sources, even some that you can generate. https://arxiv.org/pdf/2506.20057

Paper and discussion history. A mention is not an endorsement.

Universal pre-training by iterated random computation — HN submission

Sun, 29 Jun 2025 01:12:32 GMT

Universal pre-training by iterated random computation

37 points; 6 discussion replies on HN. Paper and source history. A submission is not an endorsement.