Inverse scaling can become U-shaped — mentioned by gwern

Wed, 15 Feb 2023 23:06:01 GMT

I think there is a small possibility that optimal scaling as we understand it right now (scaling compute + task/data diversity + parameter count in an optimal fashion), could just emerge (or scale) consciousness. We at present do not know what leads to emergence in the many now-documented examples of emergent capabilities in large NNs (see https://arxiv.org/abs/2206.07682#google https://gwern.net/doc/ai/scaling/emergence/index ), cannot predict either their timing or their presence, and cannot even be sure that there is not 'hidden scaling' where the capability would exhibit smooth scaling if…

Paper and discussion history. A mention is not an endorsement.

Inverse scaling can become U-shaped · HackerPapers discussions

Inverse scaling can become U-shaped — mentioned by gwern