This is a special post for quick takes by J Bostock. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
How do you guys think about AI-ruin-reducing actions?
Most of the time, I trust my intuitive inner-sim much more than symbolic reasoning, and use it to sanity check my actions. I'll come up with some plan, verify that it doesn't break any obvious rules, then pass it to my black-box-inner-sim, conditioning on my views on AI risk being basically correct, and my black-box-inner-sim returns "You die".
Now the obvious interpretation is that we are going to die, which is fine from an epistemic perspective. Unfortunately, it makes it very difficult to properly think about positive-EV actions. I can run my black-box-inner-sim with queries like "How much honour/dignity/virtue will I die with?" but I don't think this query is properly converting tiny amounts of +EV into honour/dignity/virtue.
If you don't emotionally believe in enough uncertainty to use normal reasoning methods like "what else has to go right for the future to go well and how likely does that feel", or "what level of superintelligence can this handle before we need a better plan", and you want to think about the end to end result of an action, and you don't want to use explicit math or language, I think you're stuck. I'm not aware of anyone who has successfully used the dignity frame-- maybe habryka? It seems to replace estimating EV with something much more poorly defined whic... (read more)