The Incompatibility of a Utility Indifference Condition with Robustly Making Sane Pure Bets Summary It is provably impossible for an agent to robustly and coherently satisfy two conditions that seem desirable and highly relevant to the shutdown problem. These two conditions are the sane pure bets condition, which constrains preferences...
Note: This post was pasted without much editing or work put into formatting. I may come back and make it more presentable at a later date, but the concepts should still hold. Relative abstracted agency is a framework for considering the extent to which a modeler models a target as...
Yes, the point of the proof isn't that the sane pure bets condition and the weak indifference condition are the be-all and end-all of corrigibility. But using the proof's result, I can notice that your AI will be happy to bet a million dollars against one cent that the shutdown button won't be pressed, which doesn't seem desirable. It's effectively willing to burn arbitrary amounts of utility, if we present it with the right bets.
Ideally, a successful solution to the shutdown problem should violate one or both of these conditions in clear, limited ways which don’t result in unsafe behavior, or which result in suboptimal behavior whose suboptimality falls within well-defined bounds. Rather than guessing-and-checking potential solutions and being surprised when they fail to satisfy both conditions, we should look specifically for non-sane-pure-betters and non-intuitively-indifferent-agents which nevertheless behave corrigibly and desirably.
If we implement your example, the AI is willing to bet at arbitrarily poor odds that the on switch will be on, thus violating the sane pure bets condition.
You can have particular decision problems or action spaces that don't have the circular property of the Northland-Southland problem, but the fact remains that if an AI fulfills the weak indifference condition reliably, it must violate the sane pure bets scenario in some circumstances. There must be insane bets that it's willing to take, even if no such bets are available in a particular situation.
Basically, rather than thinking about an AI in a particular scenario, the proof is talking about conditions that it's impossible... (read more)
It is provably impossible for an agent to robustly and coherently satisfy two conditions that seem desirable and highly relevant to the shutdown problem. These two conditions are the sane pure bets condition, which constrains preferences between actions that result in equal probabilities of an event such as shutdown, and the weak indifference condition, a condition which seems necessary (although not sufficient) for an agent to be robustly indifferent to an event such as shutdown.
Suppose that we would like an agent to be indifferent to an event P, which could represent the agent being shut down at a particular time,... (read 2507 more words →)
Note: This post was pasted without much editing or work put into formatting. I may come back and make it more presentable at a later date, but the concepts should still hold.
Relative abstracted agency is a framework for considering the extent to which a modeler models a target as an agent, what factors lead a modeler to model a target as an agent, and what sort of models have the nature of being agent-models. The relative abstracted agency of a target relative to a reasonably efficient modeler is based on the most effective strategies that the modeler uses to model the target, which exist on a spectrum from terminalizing strategies to simulating... (read 1221 more words →)
After hearing the problem, the question I asked myself was: at what odds would I bet that the coin came up heads? And the answer is that I would have a neutral expected return betting at 2:1 odds. This lines up with the Bayesian answer of P(heads) = 1/3.
I strongly disagree that this was the point of this in TWC and would be highly surprised if Eliezer agreed with you. For one thing, the parties involved in nonconsensual sex in TWC seem to be having a perfectly fine time. I also wouldn't be surprised if someone raping an Ancient such that they have a terrible awful no-good time would fall under some other crime and still get the perpetrator arrested.
Conjure an IQ test and take it, obviously. My IQ when dreaming ranges from greenish-purple to twelveteen o'clock.
-Eliezer Yudkowsky trims his beard using Solmonoff Induction.
-Eliezer Yudkowsky, and only Eliezer Yudkowsky, possesses quantum immortality.
-Eliezer Yudkowsky once persuaded a superintelligence to stay inside of its box.
"Different Minds (You're Concepts Formed Differently From Mine)" should probably be "Different Minds (Your Concepts Formed Differently From Mine)."
The Philosopher's Polar North (can also be translated as The Philosopher's Apex) [Nhato Remix]
https://www.youtube.com/watch?v=pc4zZ43R9o0
Turn on captions to see the lyrics and their English translation. The song says a lot about searching for truth and knowledge that I find powerful.
Some excerpts from the translated lyrics:
Accepting even those facts I have denied, ＜before time melts my memories away＞
I absentmindedly lift “truth” from an uneven distribution <as a god might guide fate＞
Even my current knowledge is still uncertain, swaying,
so with my current knowledge I should be able to keep seeking
When I break out of this shell I’ll reach out over everything
Everything in this world is in my mind; everything, is being drawn in
Everything that... (read more)
I think this might be a decent example of "rationalist" music. In particular, the lyrics communicate the value of seeking knowledge and discerning truth. There are parts that I disagree with, but overall I think it's pretty great for a Touhou soundtrack cover made by non-rationalists. Turn on captions to see the lyrics and their English translation.
The Philosopher's Polar North [Nhato Remix]
https://www.youtube.com/watch?v=pc4zZ43R9o0
I'll paste the translated lyrics here:
Accepting even those facts I have denied, ＜before time melts my memories away＞
I absentmindedly lift “truth” from an uneven distribution <as a god might guide fate＞
Even if all these impractical theories only represent my disbelief of the status quo,
there is no “normal” that... (read 393 more words →)