The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Seems like the king could still understand with a decent explanation, particularly if he bothers to ask about the effects before using it.

Agreed; the alignment plan sketched here skips over why alignment for a merely-human agent should be a lot easier than for a superhuman one, or instruction-following should be easier than value alignment. I think both are probably true, but to a limited and uncertain degree. See my other comment here for more.

I agree with your claim as stated; 98% is overconfident.

I have in the past placed a good bit of hope on the basin of alignment idea, although my hopes were importantly different in that they start below the human level. The human level is exactly when you get large context shifts like "oh hey maybe I could escape and become god-emperor... I don't have human limitations. If I could, maybe I should? Not even thinking about it would be foolish..." That's when you get the context shift.

Working through the logic made me a good bit more pessimistic. I just wrote a post on why I made that shift: LLM AGI may reason about its goals and discover misalignments by default.

And that was on top of my previous recognition that my scheme of instruction-following, laid out in Instruction-following AGI is easier and more likely than value aligned AGI, has problems I hadn't grappled with (even though I'd gone into some depth): Problems with instruction-following as an alignment target.

Could this basin of instruction-following still work? Sure! Maybe!

Is it likely enough by default that we should be pressing full speed ahead while barely thinking about that approach? No, obviously not! Pretty much nobody will say "oh it's only a 50% chance of everyone dying? Well then by all means let's rush right ahead with no more resources for safety work!"

That's basically why I think MIRIs strategy is sound or at least well-thought out. The expert pushback to their 98% will be along the lines of "that's far overconfident! Why, it's only [90%-10%] likely!  That is not reassuring enough for most people who care whether they or their kids get to live. (and I expect really well-thought-out estimates will not be near the lower end of that range).

The point MIRI is making is that expert estimates go as high as 98% plus. That's their real opinion; they know the counterarguments. 

I do think EY is far overconfident, and this does create a real problem for anyone who adopts his estimate. They will want to work on a pause INSTEAD of working on alignment, which I think is a severe tactical error given our current state of uncertainty. But for practical purposes, I doubt enough people will go that high, so it won't create a problem of neglecting other possible solutions; instead it will create a few people who are pretty passionate about working for shutdown, and that's probably a good thing.

I find it's reasonably likely that the basin of instruction-following alignment you describe won't work by default (the race dynamics and motivated reasoningh play a large role), but that modest improvements in either our understanding and/or the water level of average concern and/or the race incentives themselves might be enough to make it work. So efforts in theose directions are probably highly useful.

I think this discussion about the situation we're actually in is a very useful side-effect of their publicity efforts on that book. Big projects don't often succeed on the first try without a lot of planning. And to me the planning around alignment looks concerningly lacking. But there's time to improve it, even in the uncomfortably possible case of short timelines!

Oh and I should emphasize that if you stay in shape most of physically attractive is handled. Nice work!

Some cultures and people care about fashionable fancy clothes and gear more than others. I like to avoid the ones that are most materialistic.

I have ideas, as a fellow cheapskate!

Get a nice sun hat; Sun Day and Real Deal and Barmah all have snazzy looking widebrims (with the critical wire brim for shaping). I've been hat shopping for years even though I don't wear them much and still don't appear to be. These are barely more than the "practical" shapeless outdoor gear hats - and they work as well by most standards.

Black gaffers tape works better for most purposes than duct tape and hides instead of advertises your diy stylings.

The issue of looking like a cheapskate with repaired or worn gear and clothes is separate. Having near-new used gear and clothes instead of near-dead is only a little more expensive and does send a different message about your interest and ability to have money and take care of yourself.

Beanie in pocket is a dilemma I've faced. I unzip my jacket sometimes but that one I don't have a good solution for. Maybe in a bag? A snazzy or unique bag instead of a battered and dirty one is another cheap way to improve your curb appeal.

All in all I think you're doing great to just spend a bit of time thinking about this stuff. It's been useful for me to review it too. There low hanging fruit. It's not all or none.

I think sophistry was originally a philosophical tradition that was heavily criticized for focusing on style over substance.

Interesting points on the not-but form, I'll try to try it!

I hear you, but I'm not sure there's much tradeoff. even if appearance doesn't matter much for you outside of a romantic partner.

Doing small amounts of it is super easy. It doesn't require expensive clothes for instance, just cheap knockoffs of things people like or used versions. I guess being physically attractive in the important sense of physically fit is a large amount of effort, but that mostly pays dividends by improving your mood and energy levels in proportion to the effort. Hard to do and I struggle making it a habit, but in theory it's pretty likely a net win to spend a little time on exercise and a little discomfort and mental effort on not eating too much.

Thank you, corrected! I must've grabbed it from and misremembered from where Steve discussed it in his excellent Social status posts.

I like your emphasis on good research. I agree that the best current research does probably trade 1:1 with crunch time.

I think we should apply the same qualification to theoretical research. Well-directed theory is highly useful; poorly-directed theory is almost useless in expectation.

I think theory directed specifically at LLM-based takeover-capable systems is neglected, possibly in part because empiricists focused on LLMs distrust theory, while theorists tend to dislike messy LLMs.