Answers: (1) Not one I would trust with my future, no. (2) I would be very surprised if humans managed to build a FAI without being able in principle to reliably judge the relative value of different scenarios. (3) I would be extremely surprised if the things we currently call human values were not contradictory, even if only because (a) they're all underspecified, hence the first two questions, and (b) different humans have different values that really do conflict.
In your three scenarios, I'd say all of them are likely far better than we have any right to expect and would count as a positive singularity in my estimation, although there are enough unspecified details that could sway me otherwise.
For me the most glaring fault in them, especially (2) and (3), is that they prescribe a single kind of future that all humans somehow agree on or are persuaded/coerced to agree to. In this sense they make me feel a bit like I felt when I read Friendship is Optimal - not anything I'd deliberately set out to build as I am now, but something I'm capable of valuing and appreciating.
Also, for (1) the phrase "immortality turns out to be impossible" means very different things in sub-scenario (a) where life extension of humans beyond 120 years is impossible, vs (b) lifespans can be extended many times, possible to millions of subjective years or more, but we can't escape the heat death of the universe. VHEM seems much more understandable to me in the latter than the former world, if making new people would shorten the lifespan of every existing person by spreading resources thinner.
This seems important to me too. I have some hope that it's at least possibly deferrable until post-singularity, e.g. have the AI let everyone know it exists and will provide for everyone's basic needs for a year while they think about what they want the future to look like. Stuart Armstrong's fiction Just another day in utopia and The Adventure: a new Utopia story are examples of exploring possible answers to what we want.
-
I'm quite new to the AI alignment problem - have read something like 20-30 articles about it on LW and around, aiming for the most upvoted - and have a feeling that there is a fundamental problem that is mostly ignored. I wouldn't be surprised if this feeling was wrong (because it is either not fundamental, or not ignored).
Imagine a future world, where a singularity occurred, and we have a nearly-omnipotent AI. AI understands human values, tries to do what we want, makes no mistakes - so we could say that humanity did pretty well on the AI alignment task. Or maybe not. How do we find out?
Let's consider a few scenarios:
Now, what mark would you give humanity on the "AI alignment" task in each of the scenarios? Is there any agreement among AI alignment researchers about this? I would be surprised neither by AAA nor FFF - despite the fact that I didn't even touch the really hard problems, like spawning zillions of ems.
I have a feeling that such issues don't get the attention they really deserve - we all happily agree about being turned into paperclips, creation of smiley-faces or mindcrime, and hope the answer to the rest of the problems is "utilitarianism".
So, the question(s).