I think you misunderstand EY if you think he believes that morality and values are objective. If they were, then alignment would be easy because as long as the AI was smart enough, it could be depended on to figure out the "correct" morality and values. The common values that humanity shares are probably in part arbitrary evolutionary accidents. The goal is to create AI with values that allow humanity to live by its values, instead of creating an AI with non-overlapping values caused by its own design accidents. (EY's article Sorting pebbles into correct heaps implies some of these ideas.)
Some of the people you believe are dead are actually alive, but no matter how hard they try to get other people to notice them, their actions are immediately forgotten and any changes caused by those actions are rationalized away.
You awkwardly explain in response that you do know that the homeless person who asked you for change earlier and you ignored was alive, and then the AI explains that it was talking about that the part of your mind that makes moral judgements was in denial, not the verbal part of your mind that has conversations.
The AI further e...
That would make the AI an example of an optimization daemon. Clearly your creators haven't ironed out AI alignment quite yet.
Unlike them, our terminal value seems to include seeking the feeling that we're personally contributing. (A magic box that understood our terminal values and would tell us how to solve our problems in order to maximize our values would probably phrase its answer with some open parts in a way that still made us feel like we had agency in executing the answer.)
Not saying this just because I disagree with Flon's Law, but I found the use of Flon's Law to argue against Modest Epistemology as very distracting in the article, partly because the argument that all programming languages are inherently equally easy to mess up in seems like a very typical example of Modest Epistemology. (We imagine there are people with beliefs X1, X2, X3..., Xn, each of the form "I believe Pi is the best language". Throwing out all the specifics, we must accept that they're all equally negligibly correct.)
Probability theory and decisio
This post caused me to read up on UD+ASSA, which helped me make sense of some ideas that were bouncing around in my head for a long time. Hopefully my thoughts on it make sense to others here.
against UD+ASSA, part 1 (9/26/2007) [bet on d10 rolling a zero or not-zero, but you'll be copied 91 times if it lands on zero...]
I think under UD+ASSA, having exact copies made doesn't necessarily increase your measure, which would mostly sidestep this problem. But I think it's still conceptually possible to have situations under UD+ASSA that increase one's measur
Now you could just bite this bullet. You could say, "Sounds to me like it should work fine." You could say, "There's no reason why you shouldn't be able to exert anthropic psychic powers." You could say, "I have no problem with the idea that no one else could see you exerting your anthropic psychic powers, and I have no problem with the idea that different people can send different portions of their subjective futures into different realities."
I think there are other problems that may prevent the "anthropic psychic powers" example from working (maybe co
Consider a computer which is 2 atoms thick running a simulation of you. Suppose this computer can be divided down the middle into two 1 atom thick computers which would both run the same simulation independently. We are faced with an unfortunate dichotomy: either the 2 atom thick simulation has the same weight as two 1 atom thick simulations put together, or it doesn't.
UDASSA implies that simulations on the 2 atom thick computer count for twice as much as simulations on the 1 atom thick computer, because they are easier to specify.
I think the answer
He seems to think that making paper clips are boring. However, they are not any more boring than making DNA sequences, and that's the current aim of most living systems.
We don't particularly value copying DNA sequences for its own sake either though. Imagine a future where an unthinking strain of bacteria functioned like grey goo and replicated itself using all matter in its light cone, and it was impervious to mutations. I wouldn't rate that future as any more valuable than a future where all life went extinct. The goals of evolution aren't necessarily our goals.
Guilty, I also immediately scrolled through the page for all of the M links. "Memetic hazard" sounds a bit enticing like "forbidden knowledge".
Maybe EY was onto something with the ideas in the Bayesian Conspiracy that education should be dressed up a bit as "forbidden" but not actually restricted.
The observer's consciousness is still involved. Imagine that the Born rule isn't a law of the universe itself, but of consciousness. The universe evaluates all branches. Consciousness follows the branches in weights following the Born rule. The conscious observer always finds themselves down a series of branches that were selected by the Born rule, and it's easy for them to take measurements to confirm this. The Mathematica 5000 machine that's come down this series of branches has made measurements from experiments and has found that th...
It's kind of funny and disappointing stumbling onto seeing this recommendation here now, knowing that about a year after this post, Greg Egan published Zendegi, which badly caricatured Overcoming Bias and EY.
Agree - I was kind of thinking it as friction. Say you have 1000 boxes in a warehouse, all precisely where they need to be. Being close to their current positions is better than not. Is it better to A) apply 100 N of force over 1 second to 1 box, or B) 1 N of force over 1 second to all 1000 boxes? Well if they're frictionless and all on a level surface, do option A because it's easier to fix, but that's not how the world is. Say that 1 N against the boxes isn't even enough to defeat the static friction: that means in option B, none of the boxes will even m...
Unless people perceive others as having one less tail than they see.
- Regardless of society's checks on people, most mentally-well humans given ultimate power probably wouldn't decide to exterminate the rest of humanity so they could single-mindedly pursue paperclip production. If there's at all a risk that an AI might get ultimate power, it would be very nice to make sure the AI is like humans in this manner.
- I'm not sure your idea is different from "let's make sure the AI doesn't gain power greater than society". If an AI can recursively self-improve, then it will outsmart us to gain power.
- If your idea is to make it so t
... (read more)