This is a special post for quick takes by Zach Stein-Perlman. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
I'm worried that people won't reflect well and so (1) they'll decide to use the cosmic endowment for something mediocre and/or (2) they'll have a catastrophic failure in some of {decision theory, bargaining, dealing with threats, reasoning about simulators, acausal trade (including ECL etc.)}. Some angles:
- Current AI systems are way better at science/engineering than philosophy. Maybe even ASI won't solve philosophy (in particular, maybe there's not a sufficiently good training signal to train super-philosophers via ML).
- By default almost everyone is and will be overconfident in their intuitions on philosophy/metaethics/axiology. By default almost nobody will "suspect that they are confused about morality, and are eager or willing to apply reason and deliberation to find out what their real values are, or to correct their moral beliefs."
- Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. And it's necessary.
- It may be natural for powerful people to use AIs for instrumental stuff (acquiring power, inventing technology, building industrial capacity) but not terminal stuff (axiology, etc.). Some angles:
- If they asked an enlightened philosophy ASI about its views and it replied straightforwardly, they would think it sounded crazy.
- In general the ASIs say crazy-sounding things about cosmology, religion, etc.
- They've seen other people "reflect" with "help" from ASIs and they think those people now have crazy views, talk about living in a simulation, etc. and they don't want to end up like that.
- @habryka probably disagrees. Idk, it's so easy for me to imagine that Elon (or almost anyone else) would listen/defer to AI on the instrumental stuff but not reflect on philosophy/axiology.
- Maybe there will have been a period where superpersuaders went around getting people to adopt new ideologies—maybe even by saying true but filtered things—and so people develop strong defenses against new ideologies.
- Less importantly imo, maybe there has historically been a correlation between [competence / ability to gain power] and [values / reflective capacity] and this has driven moral progress in the past, but in the future everyone will be equally competent. h/t Will MacAskill.
- Maybe there won't be consumer demand for AI assistants who try to say true things on philosophy, as opposed to pleasing the user.
Weak counterevidence: I asked them directly and both GPT-6 and Sonnet 5.5 reported high credence in atheism.Wait, no, after logging out, GPT-6 gave a lower figure and Sonnet 5.5 refused. This is evidence for pleasing-the-user. And I just remembered Frontier models state different decision theory preferences depending on who's asking; the effect is massive.
- It may be disastrous to learn certain facts before you've learned certain other facts or taken certain actions, especially for [bargaining / threats / decision theory] reasons. E.g. avoid learning about threats or commitment races until you're prepared (e.g. for illustration, solving decision-theoretic and empirical/cosmological problems such that you respond to acausal threats correctly, and implementing SPIs). My guess is these risks are pretty easy to avoid if you try—e.g. when having AIs research decision theory or acausal interactions, you get a lot of value from simply having the AIs think for a long time in a box and then only report results and recommendations that they're pretty sure are harmless—but by default some people won't try.
- And for one, you might want to make certain deals/trades that depend on uncertainty (e.g. to trade resources in worlds that are low-stakes-for-your-values-relative-to-others' for resources in worlds that are high-stakes-for-your-values-relative-to-others'). Updatelessness may (or may not) solve this problem; it doesn't solve all problems in the parent bullet.
- (Not to mention that people might just not converge; other people might not value what I value, upon ideal reflection.)
- (Probably unknown failure modes; there's little reason to believe we know them all.)
Hmm, I started this post without @Wei Dai in mind, but I seem to just be repeating stuff he's been saying for years. There are other things Wei Dai has talked about and some of them I don't yet grok, so maybe some of them belong on this list too...
1Showing 3 of 7 replies (Click to show all)
1
1
Ω340
I think if you have some people who get the subjective time then I am optimistic that will result in them giving the people who want more wall-clock time more wall-clock time (or to figure out with them some way of augmenting that they feel happy with, if it's not literal uploading).
112
Do you think people's values will converge after that much reflection? Generalizing values seems like a domain where there's an especially high degree of path-dependency. For example, if you have advanced AI systems under your control, you might seed or shape them with your initial values, and you might even use them to shape your own mind to better suit your values too. Maybe there's some sense in which this is a mistake, but if so, it seems like you'd better have figured that out before you got too far... after, the classic arguments apply and you'd find that you don't care.
Having a strong ability to shape things really changes the nature of the reflection process. Today, someone can decide on a certain lifestyle, and then maybe after several years notice that they feel depressed, ennui, or unfulfilled, connect this to the lifestyle, and change course. In a world with advanced AI systems at your beck and call, you can just make a new drug or whatever that removes/modulates those signals, and then feel perfectly fine! Maybe later you think about this whole process, and the bespoke ideology you've crafted has you shaking your head about how easily the forces of anti-ideology could have swept you astray.
And based on being on like, twitter, I'm pretty worried that these are precisely the sorts of things that many humans would attempt to do early. There seems to be a strong impulse to notice the weakest parts of your worldview and take actions which commit yourself to these specific aspects of your worldview in a way which provides a costly signal to others. I'm worried that whatever part of the reflection process is most likely to change your mind is exactly where most people will try to engineer some mechanism to avoid that outcome.
2
Probably the main difference is I don't think we have to solve philosophy/morality in full generality to get most of the value, because of moral compromise/acausal trades, as well as strategy-stealing arguments.
I will say that this is unfortunately a difficult political problem, and this is made more difficult by superpersuasion (but it might also be easier for that reason), but we don't need much competence in philosophy/morality, what we do need is political and social buy-in for moral compromise and trade (I will say there are interesting problems in decision theory/bargaining that likely do have technological solutions), but at any rate, I don't think the problems you state are as hard as portrayed.