Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic
Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, Eliezer Yudkowsky
Abstract
We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can achieve mutual cooperation in a manner that is robust, in that cooperation does not require exact equality of the agents' source code, and unexploitable, meaning that such an agent never cooperates when its opponent defects. We construct a general framework for such "modal agents", and study their properties.
arXiv comments
18 pages. Updated 2021 to indicate that bounded analogues of these results have yet to be proved
Linked by 5 identifiable commenters in 6 comments across 5 threads, from Apr 2015 to Dec 2025. Found in 1 HN submission from Jul 2014 to Jul 2014. Open submission discussions.
With an optimal way of determining fair splitting of gains like Shapley value[0] you can cooperate or defect with a probability that maximizes other participants expected value when everyone act fairly. The ultimatum game is the simplest example; N dollars of prize to split, N/2…
This is quite fun. From a very different starting point, the idea of Team Reasoning is another non-standard way of thinking about rationality, which also starts from the idea of "shared commitments": https://www.tandfonline.com/doi/abs/10.1080/1000200309853874... The paper…
You keep hammering on with the same lazy slander. Yudkowsky was well known long before he wrote the popular Harry Potter fanfic, which incidentally is pedagogical / allegorical. Because his main roles are teacher and philosopher, and philosophers do that. Here's a sampling of…
Read Eliezer himself, 4 years ago: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... The "meat" of interest here for those too lazy to click (I encourage people click through, though, really, and maybe even take a glance at https://arxiv.org/abs/1401.5577) is…
This is a relatively old manuscript, and I would recommend instead reading, or at least first reading: * http://intelligence.org/files/ProblemClassDominance.pdf * http://arxiv.org/abs/1401.5577 (The original TDT manuscript occurred via pleas for 'something, anything to be…
With an optimal way of determining fair splitting of gains like Shapley value[0] you can cooperate or defect with a probability that maximizes other participants expected value when everyone act fairly. The ultimatum game is the simplest example; N dollars of prize to split, N/2…
benlivengoodView on HNParent context