Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? · HackerPapers discussions

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? — mentioned by Animats

Tue, 03 Jun 2025 21:28:20 GMT

This seems to be a funding proposal for "Scientist AI."[1] Start reading around page 21. They're arguing for "model-based AI", with a "world model". But they're vague about what form that "world model" takes. This is a good idea if you can do it. But people have been bashing their head against that problem for decades. That's what Cyc was all about - building a world model of some kind. Is there any indication there that they actually know how to build this thing? [1] https://arxiv.org/pdf/2502.15657

Paper and discussion history. A mention is not an endorsement.

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? — mentioned by esafak

Fri, 09 May 2025 13:11:32 GMT

In a preprint titled "Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?" (https://arxiv.org/abs/2502.15657), Yoshua Bengio et al propose a safer alternative to AGI through "scientist AIs" that explain but do not have agency, which they define as: "Agents observe their environment and act in it in order to achieve goals". I think non-agency is crucial to preventing catastrophes, but even without it models could induce humans to take action on their behalf, to similar ends.

Paper and discussion history. A mention is not an endorsement.