qbolec

We often laugh over human-specific "bugs" in reasoning, comparing it to a gold standard of some utility-maximizing perfect Bayesian reasoner. We often fear that a very capable AI following strict rules of optimization would reach some repugnant conclusions, and struggle to find "features" to add to guard against it. What if some of the "bugs" we are looking at, are actually the "features" we are looking for? * We seem to distinguish "sacred" and "non-sacred" values, refusing to mix the two in calculations (for example human life vs money). What if this "tainted bit", "NaN-propagation", is a security feature guarding against Goodharting leading to genocide or dissolution of social trust? What if utility is not a single real number, but instead a pair? What if the ordering is not even lexicographic, but partial? What if it's a much longer tuple? Which brings me to next point: * We often experience decision paralysis apparently unable to compare two actions. What if this is simply because the order must be partial for security reasons? An alternative explanation of this phenomenon is that we implicitly treat "wait for more data to arrive and/or situation to change in tie-braking way" as an action available to us - is that bad? * We often decide which of the two end-states A vs B we prefer based on the path leading to them, amusingly favoring A in some scenarios and B in others. What if this is because we implicitly assume that end-state contains our brain with the memory of the path leading there? Isn't this a cool feature to treat the agent as part of its environment? Or what if this is because we implicitly factor in considerations of "what if other society members would follow this kind of path, or decision-making algorithm?". Isn't this a cool feature to think about second-order effects, about "acausal trades", and to treat own software as perhaps shared with other agents? * At least some long-term stable cultures have norms requiring children to follow ad