I do think it's not good! But also, it's an important issue and you have to interface with people who aren't super principled all the time. I just don't want people to think of the AI Safety community as some kind of community of saints. I think it's pretty high variance, and you should have your guard up a good amount.
1Is this not common in politics? I thought this was a lot of what politics was about.
I have been very surprised by how non-agentic politics is! Like, there certainly is a lot of signaling going on, but when reading stuff like Decidingtowin.org it becomes clear how little optimization actually goes into saying things that will get you voters and convince stakeholders.
I do think a lot of that is going on there, and in the ranking above I would probably put the current political right above AI safety and the current political left below AI safety. Just when I took the average it seemed to me like it would end up below, largely as a result of a severe lack of agency as documented in things like deciding-to-win.
Re corporate campaigns: I think those are really very milquetoast. Yes, you make cool ads, but the optimization pressure here seems relatively minor (barring some intense outliers, like Apple and Disney, which I do think are much more agentic here than others, and have caused pretty great harm in doing so, like Disney being responsible for copyright being far too long in the US because Disney was terribly afraid of anyone re-using their characters and so tainting Disney's image).
Very low, though trending a bit higher over time. The policy-focused playbook has to deal with a lot more trickiness here than AI-2027, and you have to deal more with policymakers and stuff, but currently y'all don't do very much of the kind of thing I am talking about here.
I think this is roughly what happened when FTX was spending a huge amount of money before it all collapsed and a lot of people started new projects under pretty dubious premises to look appealing to them. I also think this is still happening quite a lot around OpenPhil, with a lot of quite bad research being produced, and a lot of people digging themselves into holes (and also trying to enforce various norms that don't really make sense, but where they think if they enforce it, they are more likely to get money, which does unfortunately work).
Yep, I agree that is a non-trivial part of it as well. A lot of the complexity was part of the optimization process itself, which could eventually be pruned away as people understood the parameters of the problem more. I agree that's a pretty different thing.
I currently work for an org that is explicitly focused on communicating the AI situation to the world, and to policymakers in particular. We are definitely attempting to be strategic about that, and we put a hell of a lot of effort into doing it well (eg running many many test sessions, where we try to explain what's up to volunteers, see what's confusing, and adjust what we're saying).
Yeah, that does sound roughly like what I mean, and then I think most people just drop the second part:
But, importantly, we're clear about trying to frankly communicate our actual beliefs, including our uncertainties, and are strict about adhering to standards of local validity and precise honesty: I'm happy to talk with you about the confusing experimental results that weaken our high level claims (though admittedly, under normal time constraints, I'm not going to lead with that).
I do not think that SBF was doing this part. He was doing the former though!
Am I just blind to this? Am I seeing it all the time, except I have lower standards what should "count"? Am I just selected out of such conversations somehow?
My best guess you are doing a mixture of:
- Indeed self-selecting yourself out of these environments
- Having a too-narrow conception of the "AI Safety community" that forms a Motte where you conceptually exclude people who do this a lot (e.g. the labs themselves), but in a way that then makes posts like the OP we are commenting on misleading
- Probably have somewhat different standards for this (indeed, a thing I've updated on over the years is that a lot of powerful optimization can happen here between people, where e.g. one party sets up a standard in good-faith, and then another party starts goodharting on that standard in largely good-faith, and the end-result is a lot of deception).
1Don't religions sort of centrally try to get you to believe known-to-be-false claims?
I agree that institutionally they are set up to do a lot of that, but the force they bring to bear on any individual is actually quite small in my experience, compared to what I've seen in AI safety spaces. Definitely lots of heterogeneity here, but most optimization that religions do to actually keep you believing in their claims are pretty milquetoast.
Are you saying that EAs are better at deceiving people than typical members of those groups?
Definitely in-expectation! I think SBF, Sam Altman, Dario, Geoff Anders plus a bunch of others are pretty big outliers on these dimensions. I think in-practice there is a lot variance between individuals, with a very high-level gloss being something like "the geeks are generally worse, unless they make it an explicit optimization target, but there are a bunch of very competent sociopaths around, in the Venkatesh Rao sense of the word, which seem a lot more competent and empowered than even the sociopaths in other communities".
Are you claiming that members of those groups may regularly spout false claims, but they're actually not that invested in getting others to believe them?
Yeah, that's a good chunk of it. Like, members of those groups do not regularly sit down and make extensive plans about how to optimize other people's beliefs in the same way as seems routine around here. Some of it is a competence side-effect. Paranoia becomes worse the more competent your adversary is. The AI Safety community is a particularly scary adversary in that respect (and one that due to relatively broad buy-in for something like naive-consequentialism can bring more of its competence to bear on the task of deceiving you).
My current best guess is that you have a higher likelihood of being actively deceived/have someone actively plot to mislead you/have someone put in very substantial optimization pressure to get you to believe something false or self-serving, if you interface with the AI safety community than almost any of the above.
A lot of that is the result of agency, which is often good, but in this case a double-edged sword. Naive consequentialism and lots of intense group-beliefs make the appropriate level of paranoia when interfacing with the AI Safety community higher than with most of these places.
"Appropriate levels of paranoia when interfacing with you" is of course not the only measure of honor and integrity, though as I am hoping to write about sometime this week, it's kind of close to the top.
On that dimension, I think the AI Safety community is below AGI companies and the US military, and above all the other ones on this list. For the AGI companies, it's unclear to me how much of it is the same generator. Approximately 50% of the AI Safety community are employed by AI labs, and they have historically made up a non-trivial fraction of the leadership of those companies, so those datapoints are highly correlated.
Huh, those are very confident AGI timelines. Have you written anything on your reasons for that? (No worries if not, am just curious).
Also, come on, this seems false. I am pretty sure you've seen Leverage employees do this, and my guess is you've seen transcripts of chats of this happening with quite a lot of agency at FTX with regards to various auditors and creditors.