This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
i often think specific capabilities projects are quite unlikely to work, and therefore not worth taking into account when coming up with my alignment approach, and also simultaneously that my alignment project is quite unlikely to work, but it's worth trying in case it does work. why this asymmetry?
i claim this is rational. the key is that upside risk and downside risk should be treated different. if i think an alignment approach has a 1% chance of working, it might still be worth spending my life on. but if i think there’s a 1% chance some capabilities technique might work that invalidates my alignment approach, i shouldn’t spend time thinking too much about that possibility.
suppose the success of my alignment approach and the success of a particular capabilities approach are uncorrelated, my alignment approach is completely ruined if the capabilities approach succeeds, and both are 1% likely to succeed. then, the delta from trying to make my alignment approach work multiplies p(doom) by 0.9901, whereas trying to make my approach succeed while also taking the capabilities approach into account only reduces p(doom) very slightly more - it multiplies p(doom) by 0.99.
the 1% is just illustrative. i think this is missing the point. you can feel free to apply whatever transformation you want to your inside view probability. suppose you think that plans that feel 99% likely to work on the inside view are actually only 1% likely to work overall. then just use my analysis for plans that feel 99% likely to work.
separately i think death with dignity as a post is wrong. it is so preoccupied with arguing against a specific kind of dumb person - one who would say "well if it's hopeless we should just do some random ass stupid gamble and play for variance" - that it prescribes something insane for the rest of us. the way to win is to do lots of uncorrelated research bets that are individually unlikely to succeed but also do no harm if they fail! that's the way surprising and novel scientific inventions have always happened in the past. (yes, i know many approaches are actually correlated. this is a skill issue)
1Andrii Vasylenko
I disagree that alignment will probably get solved by someone pursuing a direction that seems very unlikely even to them. It seems to me that the right way to do things is to figure out what the hard parts of alignment are and then to try to solve them.
[...]
I think that's because those fields had reasonably good feedback loops, and so the strategy of "try a bunch of things and see if any of them work" is generally viable. As a counterexample, Einstein's methodology was on the opposite end of the spectrum.
well, no, i think alignment will probably get solved by someone who has delusional confidence that they have the one true approach, who maybe on some intellectual level knows that it's 1% likely to succeed but feel irrationally driven to make it work, embedded inside a system that is not delusional and able to assess alignment approaches rationally. most such people will be wrong in their delusion. perhaps the world will be destroyed by people with delusional confidence embedded inside systems that are also delusional. but it's very hard to truly devote yo... (read more)