An offer you can't (rationally) refuse: systematically exploiting utility-maximisers with malicious gambles.

Abstract

Decision theory aims to provide mathematical analysis of which choice one should rationally make in a given situation. Our current decision theory norms have been very successful, however, several problems (such as Pascal’s Wager, the St. Petersburg Paradox, and Pascal’s Mugging) have proven vexing for standard decision theory. In this paper, I show that these problems all share a similar structure and identify a class of problems which decision theory overvalues. I demonstrate that agents who follow current standard decision theory can be exploited and have their preferences reordered if offered decision problems of this class. I show that preference reordering is a serious problem, which motivates my search for a decision theory which is immune to exploitation. I find Dr. Nick Smith’s theory of Rationally Negligible Probabilities cannot be exploited in this way and discuss why agents should adopt it.