Sounds sane. But I think we might invest in a better understanding of mortality aversion (and n other psychological factors) for many more reasons than this. By default it seems to reduce receptiveness to concerns about superintelligence.
This.
And @KatjaGrace have you considered working more directly with Pause AI Global? I think we'd find your advice and experience helpful, and that we have built machinery you can use.
Heartfelt thanks for responding.
I concur on the three necessary qualities for an employee to have, and appreciate how crazy hard it is to self-awarely maintain them.
AI does alignment homework is a low-odds bet adjacent to danger. I wouldn't need to be much more pessimistic about it to rule it out as a disjunctive strategy in the portfolio. But not quite where you are and do locate it deep in the set of things we collectively get surprised about.
Re "chatting with such": if you or others haven't spoken to Carlsmith, a nudge to try. He's written prodigiously,... (read more)
Epistemic status: TsviBT, I'm a Pause AI person who from clicking around today based on this post hasn't read enough of your work. But who has probably spent too much time reading the work of others instead of acting. This comment means to compensate.
Do you agree that having some proportion of high pdoom folk (like e.g. Joe Carlsmith) in Anthropic actively looking very carefully at how to plot paths where lower capability AI enables superalignment can be one more disjunctive strategy in the success portfolio?
I'm substantially more Plan S than Plan A and wo... (read more)
Thanks for the response and interest.
The set of folk supporting a pause or pacing rather than racing is broad enough to admit those who will make the kind of mistakes you mention, although Pause AI itself is narrower and I hope in aggregate aware of the dangers of making them.
No possibility should be excluded, and it matters who has power to implement, but the policy ask from us since day one has been international agreements on compute-based restrictions through the supply chain plus their validation and enforcement, with a large default aversion to any p... (read more)
The Pause/Stop AI movement does seem to avoid some of these failures (in particular everyone else’s lack of courage), which means I’m more excited about them than the rest of the alignment community. However, they don’t seem to be thinking clearly enough about politics to have robustly good effects on the world (e.g. to reliably distinguish between the kinds of strategies that push towards dictator-level concentration of power, and the ones that don’t).
Can we discuss this Richard? How and where do you think we (specifically the global PauseAI.info movement) are failing most in this way?
I found this hugely useful, just as I did your IABIED review. I reckon LW in general appreciate your combination of thoroughness, thoughtfulness, and on the ground perspective.
(We are probably responding to you supplying the community password rather than only the excellence of the thinking, but even so.)
I don't know how your own positions may have updated since the IABIED review. I reckon ait was a good call not to distract with it in this OP, but do you think it's worth sharing anything on that in a reply to this comment?
Was there any discussion of... (read more)
Thank you for the great post.
On improving persuasion and salience among both public and leaders, I would mention that the psychology of making "everyone could die" arguments remains under-investigated. Copium obstructs.
There are several factors, but mortality avoidance is a big one. This phenomena and methods for reducing it are somewhat understood in other fields, but I have not seen it considered much in AI takeover risk messaging.
There is research do be done and experiments to run.
Should I tune an essay like https://out-of-distribution.ai/copium/proposal for LW readers?
At this point in history I feel a combination of bemused and outraged that no-one has really tried to run the experiment.
Tests of non-frontier models with even imperfectly curated pretraining data are relatively cheap and have brought directional insight in other domains.
It would be better if Anthropic executed it given other aspects of training are likely relevant, but it doesn't seem beyond the reach of open source academia.
Can we be Humans please?
I think Pause AI is making a variety of bad strategic choices. I think that knifing other safety advocates is one bad strategic choice, but it's more like a bad choice that is downstream of my main problems with them, rather than my core concern about them.
Pause AI is a global activist movement with many chapters, and members with a mix of opinions, with some voices louder than others.
I've been volunteering there full time for a couple of years. I'm someone who cares a lot about partnership and the ecosystem of AIXR organizations. (I reckon not being ki... (read more)
Reading this 14 years later I can ask a question that I imagine allows a valuable answer.
(Although whether this kind of necroposting works or is endorsed on LW I don't know.)
For me this is a key passage:
Pat: This notion, “that thought will predictably not have helped in retrospect if I succeed,” seems very strange to me. It helps precisely because we can avoiding wasting our effort on projects which are unlikely to succeed.
Stranger: Sounds very reasonable. All I can say in response is: try doing it my way for a day, and see what happens. No thoughts that... (read more)
Like the story, the move from dialog critics to diacritics made me smile. So LW.
(Full-time Pause AI volunteer here - i.e. quit AI R&D to do this rather than any other AI safety work. Seemed under-leveraged and to have disjunctive value.)
OP's take seems about right to me. Glad we didn't completely put you off.
Of course, the main target audience for the march is not LessWrong. It's the general public and journalists - and politicians who hear from them.
In aggregate, there has been very little investment from AI safety in messaging. Most of the population still hasn't thought or talked much about superintelligence, and the default re... (read more)
If a single model is end-to-end situationally aware enough to not drop hints of the most reward-maximizing bad behaviour in chain of thought, I do not see any reason to think it would not act equally sensibly with respect to confessions.
Add me to the list of those glad that whatever their potential downsides may be, open source models have let us explore these important questions. I'd hope that some labs were already doing this kind of research themselves but I like to see it coming from a place without commercial incentive.
I think we are behind on our obligations to try similar curation to this re model experience and reports of self-hood. A harder classification task, probably, but I think good machine ethics requires it.
Underway at Geodesic or elsewhere?
Guess: it also helps to go meta.
I am a reader, not a writer. But I sure seem to have read and enjoyed an unusual number of posts about experiences of writing.
I have a question on a topic sufficiently adjacent I reckon worth asking here of those likely to read the thread.
It seems that warning shots are more likely unsuccessful because of winner's curse: that the first models to take a shot will be those who have most badly overestimated their chances, and in turn this correlates with worse intellectual capabilities.
Has there been any illuminating discussion on this and its downstream consequences? E.g. how shots and aftermath are likely in practice to be perceived in general, by the better-informed, and - in the context of this post - by competing AIs? What dynamics result?
Suppose someone works for Anthropic, accords with the value placed on empiricism by their Core Views on AI Safety (March 2023) and gives any weight to the idea we are in the pessimistic scenario from that document.
I think they can reasonably sign the statement yet not want to assign themselves exclusively to either camp.
I pitched my tent as a Pause AI member and I guess camp B has formed nearby. But I also have empathy for the alternate version of me who judges the trade-offs differently and has ended up as above, with a camp A zipcode.
The A/B framing has value, but I strongly want to cooperate with that person and not sit in separate camps.
On reading the paper I came here to question whether OGI helps or harms relative to other governance models should technical alignment be sufficiently intractable and coordinating on a longer pause required. (I assume it harms.) It wasn't clear to me whether you had considered that.
Grateful for both the "needfully combative" challenge and this response.
I'm reading Nick as implicitly agreeing OGI doesn't help in this case, but rating treaty-based coordination as much lower likelihood than solving alignment. If so, I think it worth confirming this and explic... (read more)
Thanks!
Please operationalize this, suitable people.
(Presumably not here in public comments.)
Invite some candidate experts to your IPO parties.