This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
People with p(doom) > 50%: what concrete empirical achievement on current or near-future models, if any, would bring your p(doom) under 25%?
Answers could be anything from "the steering vector for corrigibility generalizes surprisingly far" to "we completely reverse-engineer GPT-4 GPT-6 and build a trillion-parameter GOFAI without any deep learning".
Great question! This is valuable meta-problem work; it was a valuable exercise. Here's my answer, after some consideration, for what would make me a lot more optimistic:
Seeing decision-makers speak clearly and sanely about simple core alignment logic - see below
Seeing more other people speak clearly and sanely, calling for global cooperation
Technical alignment progress including
Improved monitoring/alignment in agent scaffolding - see below
Recommitment to faithful CoT and confession training
Advances in mechinterp
Better scenarios exploring alignment during ... (read more)