When I read Holly's Twitter feed, I get why someone would have a gut reaction like "this is an unhinged person who's stooping to the level of personal attacks on people who seem nice and are just trying to do their best". I have my disagreements with the degree to which she bases her public statements about people's character on her own subjective judgment, as opposed to explicitly reasoned-out judgments. I gave her this criticism when she came on Doom Debates, and we still disagree about it.
That said, I also often see Holly publicly post moral character judgments that I think are reasoned, and that I personally agree with. The reasoning often takes this form:
- A person is currently helping push the frontier of AI capabilities, despite runaway ASI being a high imminent existential risk.
- Ergo that person is doing morally bad actions.
- Ergo we judge them as a morally bad actor.
I think this is a straightforwardly reasoned judgment, assuming one agrees with the world-model premises like "if anyone builds ASI anytime soon, everyone dies".
Holly has been one of the first and still only voices who is simultaneously savvy about the underlying world model and the culture of people who share it, a...
10
1I'm quite unsure what to think, but I like this comment.
A person is currently helping push the frontier of AI capabilities, despite runaway ASI being a high imminent existential risk. Ergo that person is doing morally bad actions. Ergo we judge them as a morally bad actor.
I think this doesn't actually follow. It does matter whether they know what they're doing is bad. If they know and continue, they're a morally bad actor.
Now, "Do they know?" is a very messy question to answer, and a principal Sinclair's Razor strategy is to self-deceive. Self-deception, especially in this context, is morally bad too. Over time, it does seem more and more like everyone is either self-deceiving or ok with their actions being bad, and either way is a morally bad actor.
As you perhaps suggest here...
As I mentioned at the beginning of my comment, I also see her as going too far in the "publicly judging people" direction by sometimes doing it on a subjective basis, i.e. without adequately reasoning it out in enough detail to lead a neutral reader to her moral conclusion. On Doom Debates, she didn't even object to my characterization of her doing this.
...there's an important difference between it ...
2
2Re: “unhinged.“ If/when the general public starts figuring out the plans of capabilities researchers - destroying everyone‘s jobs, creating a replacement for humans, destabilizing the fabric of human life, potentially installing a global government to manage human affairs for all time - I think the response will be much more “unhinged“ than anything Holly’s ever done. It might be worth updating on this “unhinged“ness now rather than later.
As for how to persuade such researchers across a belief gap. I think a useful intuition pump might be, “how would I convince a dangerous gain of function virologist to stop what they’re doing?” They will have arguments that are likely to be just as galaxy brained, or they fundamentally disagree with you in some way you didn’t expect, or they just never heard about it being existentially bad, or they don’t care.
I would push back against a norm of “if you can be out argued, you must submit.” Argumentative ability does not necessarily correlate with being right; if a random person can offer 5 plausible seeming justifications for why you should sacrifice your life, and you can’t come up with a response other than “I don’t like this,” that doesn’t mean your life should be forfeit!
3
2It is important to acknowledge that the relationship has broken down, and this seems to me like a positive update about the health of PauseAI. Elmore has repeatedly been unable to build alliances; I recall after the first protest co-organized with Liron Shapira, who did a great job with speeches and chants and taking the microphone, she declared she would not work with him again (edit: Liron reports they made up after this downthread), and she has managed to alienate many of her would-be allies.
I must however critique your principles:
We do not attack their character. We do not make humans the problem. The problem is their actions. Attacking people’s character is not aligned with our fundamental commitment to nonviolence, neither in principle, nor in the outcomes it generates. We will not win by vilifying the people we’re trying to bring to the table. We will also not build the wholesome, sustainable, broad-based, vibrant movement the world needs by running on an engine fuelled by personal drama and shaming.
I agree that prejudicial and ignorant accusations about people's character is very costly, and I can believe that you do not believe yourself capable of maintaining the ethics of...
1But it is critically important to talk about their character on the large stage.
I think maybe individuals should speak of people's characters, but orgs shouldn't?
When orgs say things, they become party lines, large numbers of people parrot. And because no person can know the mind or inner life of anyone else, it seems particularly bad for claims about people's character to become party lines.
I definitely agree that we must be able to talk publicly about people's track records, and trustworthiness, but that's a tricky thing to do well.
I think I would prefer if orgs stick clearly commenting on observations (eg "OpenAI's structure was explicitly designed so that the board could fire the CEO, but when that mechanism was triggered, the Sam remained CEO", and "Anthropic said XYZ, in their original RSP, but then later walked that back"), but leave judgements up to individuals (eg "I don't trust Sam Altman", "I think Dario is mainly motivated by power-seeking").
Pause AI can verify the observations. Pause AI can't really speak to the judgements of all it's members, who surely have all kinds of different views (unless they do a survey or something, in which case, the results of the survey count as a verifiable observation).
1I recall after the first protest co-organized with Liron Shapira, who did a great job with speeches and chants and taking the microphone, she declared she would not work with him again
I suspect there is more to this particular story, given her recent appearance on Doom Debates: https://youtu.be/wOtgPgb6lGk?si=1Ajfix3y4nGz7zVF
RE character attacks: not trusting Sam Altman to keep commitments based on prior actions isn't what's at issue here. See above video link for an exploration of what appears to be the crux (would normally give a tl;dr but this is a place where hard-to-summarize nuance is relevant).
1We had a conflict for like a month in 2023 relating to an early PauseAI protest. I'd say we've avoided conflict since then :)
1I... was not aware of that. That's genuinely a new low if she did it, I'd have thought she had more self-respect than that. (Curious to see a link.)
Myself, I had assumed that this meant Elmore was regularly opening (i.e. ending) discussions with flat statements that people are evil, hollow, devoid of care for their fellow man, self-serving, craven, and more. And, while there's definitely some degree of that, that just isn't required to commit an atrocity. The banality of evil, and all that. As well as awful people, there are also a lot of nice people at AI labs who you'd be friends with in other worlds. If she is doing this with prejudice, which, I basically believe she does, then she's just muddying the waters, adding tons of cruelty, and lowering the standards for ethical behavior in the domain of public discussion (where movements like PauseAI substantially operate).
1No, it is not "petty infighting"; so far as I currently know, PauseAI are okay people, and they have legit reasons for needing to sever ties with "PauseAI US".
5
2To what extent are you endorsing the reasons they gave in this post, versus reporting knowledge of other legit unspecified reasons?
From my perspective, the reasons given in this post are bad enough to cast doubt on any other reasoning that PauseAI is doing. In particular:
Attacking people’s character is not aligned with our fundamental commitment to nonviolence, neither in principle, nor in the outcomes it generates.
Characterizing attacks on people's character as a kind of violence is absurd—the ability to critique and evaluate people's characters is one of the foundational mechanisms which makes ethical coordination possible.
Not only is this a bad mistake in general, it is specifically one of the key mistakes that has led to the AI safety movement becoming ideologically captured, as I recount in this post.
So it seems important for you (Eliezer) to clarify the extent to which you endorse this blanket prohibition on character-based critiques.
...But we think the AI safety community is beginning to change its mind and will put its weight behind a pause if we show them this is the better, viable path. We believe that making it easy for people with different views to come round is of immens
16Yeah, FWIW, this post was a major update downwards for me on PauseAI Global. I had hope they could do much better than PauseAI US, but the vibe I am getting is that they are falling into the conspiracy attractor, which is not actually much better (and I think probably actually substantially worse) than the "become a mindkilled soldier yelling at everyone" attractor.
I do also want to put at least some blame on what is going on here on the funders in the ecosystem. It is true that SFF has blocked all funding to PauseAI because they blocked all funding to advocacy, and all funding to anything that resembles "calls for violence" according to a definition so broad as to explicitly include "jokes in which a fictional character advocates for violence". PauseAI Global was basically told they couldn't get funded because of that by SFF, and it seems like this likely contributed to PauseAI Global becoming kind of paranoid in this dumb way.
2Holly has previously called for a "Nuremberg Trial" of all AI company employees, including safety-focused researchers (here) which does seem to me to go beyond criticism and into calls for (indirect, state) violence.
What? I absolutely support Nuremberg trials for AI company employees. And I hope compared to the historical Nuremberg trials these ones actually lead to substantial convictions! How on earth is calling for criminal trials a call for violence. Of course I want people who put everyone on earth in danger to be punished for doing that, by the state, using the methods of justice on which our civilization is built on.
And we mostly get to avoid the key difficulties of the Nuremberg trials, which was that it was all operating in a foreign and international jurisdiction. I of course hope that people succeed at bringing forward legal challenges towards the AI companies for putting everyone on earth in danger, and think the legal system is one of the best tools we have for stopping what is going on in terms of AI development.
21Given that you conceded immediately that you oppose ex post facto laws, it's mostly just a semantic debate remaining—but the term "Nuremberg trials" in my mind basically means "we form an extraordinary tribunal of our friends and execute our defeated enemies for breaking retroactive laws we make." So that explains the feeling of a call for violence: the extralegal element.
For example, Julius Streicher was convicted and executed only for "crimes against humanity," a crime invented for the trial itself, on the basis that he was an anti-semitic propagandist (he founded Der Stürmer), which was not illegal at the time. So someone might think that those who advance a "Nuremberg Trial" want pro-AGI propagandists to be retroactively tried and executed.
1My (mostly off-the-cuff) answer is that [ex post facto prosecutions under a "no ex post facto prosecutions" legal framework] are a very different beast than [ex post facto prosuctions under an absence of legal framework].
I don't take Habryka's "can people stop saying this" as "this is false" but as "this is a bad thing to focus on in context".
1Any such court should only prosecute behavior that actually violated any laws/statutes/international agreements, and should not deviate from existing principles of the US justice system. Though also, common law has established a long track record of substantially coining new rules that are sufficiently important for justice to be served, as an extension of existing legal doctrine or broad common-sense principles.
So yeah, I agree. "No ex post-facto law" should be an extremely large consideration in any trial, and should majorly limit liability given the absence of a specific statute banning AGI development.
That said, I don't think the absence of any specific statute banning the development of AGI eliminates liability completely, as there are various existing statutes that a court could consider here to bring forward charges, some of which would seem quite strong to me (for example, it seems quite clear to me OpenAI should face criminal liability for the Huggingface incident, and think legal precedent for this is relatively robust, if not 100% clear cut). There are also various general purpose doctrines, like "reckless endangerment" which don't require actualized harm, and where pros...
1And yet this would not be unprecedented.
Most historians agree that the Nuremberg trials were at least in part ex post facto. I think most people would agree that the Nuremberg trials were well-deserved and necessary.
Likewise, habeas corpus is one of the standard rules of justice, and is also in the U.S. Constitution. Abraham Lincoln suspended habeas corpus in Maryland in the early months of the Civil War, so as to prevent the state from seceding and ensure that rebel brigands didn't cut off supply and comms lines between Washington DC and the rest of the Union. Lincoln's rationale for this was that the law is not a suicide pact and it's better to temporarily violate one law than to condemn the nation to suicide:
"Are all the laws, but one, to go unexecuted, and the government itself go to pieces, lest that one be violated? Even in such a case, would not the official oath be broken, if the government should be overthrown, when it was believed that disregarding the single law, would tend to preserve it?"
Do you also believe those actions were unjustified under the same principles - did they also violate the spirit of the law - or do you consider the case of frontier AI developers to be...
1Why do people keep thinking that?
People keep saying that because it's commonly understood to be a core part of the Nuremberg trials. (Claude output as an example below.)
Maybe this is a cultural difference, as I can imagine that in Germany the primary talking point about the Nuremberg trials is only that it didn't go far enough … But the key parts of calling for "Nuremberg trials" I think are that
- they charge defendants with capital crimes
- they target leaders like politicians, business executives, etc
- they retroactively introduce new crimes
- they reject the so-called "Nuremberg defense" that defendants were following orders
"New legal ground may be required"
"Standard defenses are pre-emptively rejected"
"Existing institutions are inadequate or complicit"
...yyyeah, I'm sorry, if this seems reasonable-ish to you I do not think I want you to come anywhere near political power, and I don't even work at an AI lab.
1If one called for Sabrina, the young lady who put everyone in town in danger by practicing witchcraft, to be subjected to a trial by ordeal, you wouldn't be so confused by the term and the aesthetics of the process that you'd think it meaningfully different from calling for her to be drowned or whatever.
I mean, yes, historical legal trials were barbarous. Current legal trials are mostly not barbarous. They are onerous, they are not perfect, but I generally do not consider bringing someone to court anything close to a "call for violence". If someone were to compare current legal trials to historical witch trials, they would be very confused and be using words badly.
Of course the state should then use violence appropriate, but calling for a trial is not a call for violence, calling for a trial is saying that you want the state to look into whether violence is appropriate, and that you will defer to the established justice processes on the ultimate result.
1Look, I think suing people for libel is a shitty thing to do, because libel law is kind of broken. But I of course would not remotely say that someone who is threatening to sue me with libel is "threatening me with violence".
You appear to be committing an enormous fallacy of grey. It is really extremely obvious that present justice systems are vastly vastly better than random medieval witch trials. Furthermore, a "commitment to nonviolence" which includes refusing to have someone stand trial is just extremely silly. How are you proposing to resolve people causing harm to others if not by letting them stand trial! Standing trial is the exact thing we have to avoid having to revert to individual violence.
1Look, man, I've explained why it's a joke in context, you're allowed to not find it funny if you want. But if you're ignoring the tweets where she compares me to a defender of slavery, I don't think you're engaged seriously in this discussion.
I don't want to debate this, I am tired of working on behalf of AI safety and being compared to a slaveowner for going to the same conference that dozens of other LessWrong posters were at, I am so, so tired of you folks somehow thinking the most vile thing you could say about an American is possibly okay.
2
1One of the two groups should officially rebrand as soon as possible.
You're trying to court the general public, who will not look closely enough to distinguish between the two.
1Agreed, though if I were Pause AI (the international org), I would at least want to pause and consider the wisdom and ethics of re-branding to distance themselves from Pause AI US, if letting them keep the brand is effectively rewarding (what they regard as) bad behavior.
On the other hand, in general, you should change the things that you have control over, and you don't have control over what other people and other orgs do. So if you think (reasonably!) that these two orgs need separate brands, it's good to make that happen via changing your brand.
One possibly relevant consideration is that, depending on the diversity of their funding sources, it seems likely to me that Pause AI US will just die, for lack of funding. There's already a lot of discontent among a wide base of people with Holly's behavior. I don't think it would take very much for potential funders to demur because of the bad reputation, this might turn out to be the final straw.
1We endorse pointing out harmful actions. But we condemn attacks on people’s character.
I am not a fan of Holly Elmore but surely it's absurd to make this distinction! It's perfectly reasonable to think someone's harmful actions reflect poorly on their character.
In fact, one of the best kinds of evidence we have with which to judge someone's character is their actions.
32It's perfectly reasonable to think someone's harmful actions reflect poorly on their character.
I think this is the part that's missing the point. The question isn't whether people's actions are indicative of their character; it's whether focusing on people's character is more productive than focusing on their actions.
Holly says "yes" on the basis that it moves the Overton window, generates revenue, and attracts volunteers. In contrast, Joep says "no"--I think on the basis that it's less intellectually rigorous, less inclusive, and possibly that it paves the way for violence. (#3, #9, and #1 in PauseAI's statement of values)
1Speaking entirely in my personal capacity, I noticed that direct quotes from Holly Elmore are mostly missing from this piece, so I thought I'd add one that resonated with me:
"It’s wrong to work for the AI companies and it’s wrong to defend them. It’s wrong to have friends who have a stake in disempowering or destroying humanity. This is not something we can be nice about and discuss over tea. There is no 'right' way to deliver this vital critique— the reason they don’t want to hear it is that they are doing wrong."
https://x.com/ilex_ulmus/status/2070746368910520506
I don't draw my moral lines in exactly the same places that Holly does -- I think it can be reasonable to work for an AI company if you're working almost exclusively on safety or if you have a detailed, concrete, public commitment about how you will use your leverage for safety. I genuinely value much of the safety work being done by frontier developers. I also think it's fine to attend conferences and other professional events with AI capabilities researchers and to negotiate with them -- half the point of professional events is to provide an opportunity for dialogue among people who are not personal friends.
That said, ...
2I fixed the very broken table-formatting you somehow put into our post-editor (we do support tables, but not this way). I think this might have changed the formatting in your post a bit, feel free to update and correct.
8
3I'm sure there are many valid reasons and the situation is complicated but I can't help putting this here https://www.youtube.com/watch?v=a0BpfwazhUA
3That clip is surprisingly relevant: https://hollyelmore.substack.com/p/no-the-judean-peoples-front-should
5
1
- We look forward to building more with you all, including those of you based in the US.
- PauseAI officially disendorses current PauseAI US leadership.
- We look forward to building more with you all, including those of you based in the US
Is this a typo? Did they just really want to emphasize the first message?
4
This morning, I got an email from the CEO of PauseAI. I will paste the text below. PauseAI has decided to distance themselves from PauseAI-US, with whom they share branding, but apparently not much else. This is a really confusing situation for volunteers and newcomers. I think it would be worth having a discussion to see how we can proceed in such a way that volunteers, especially in the US, are able to effectively direct their activism.
Email from PauseAI: