My name is Mikhail Samin (diminutive Misha, @Mihonarium on Twitter, @misha on Telegram).
Humanity's future can be enormous and awesome; losing it would mean our lightcone (and maybe the universe) losing most of its potential value.
My research is currently focused on AI governance and improving the understanding of AI and AI risks among stakeholders. I also have takes on what seems to me to be the very obvious shallow stuff about the technical AI notkilleveryoneism; but many AI Safety researchers told me our conversations improved their understanding of the alignment problem.
I believe a capacity for global regulation is necessary to mitigate the risks posed by future general AI systems. I'm happy to talk to policymakers and researchers about ensuring AI benefits society.
I took the Giving What We Can pledge to donate at least 10% of my income for the rest of my life or until the day I retire (why?).
In the past, I've launched the most funded crowdfunding campaign in the history of Russia (it was to print HPMOR! we printed 21 000 copies =63k books) and founded audd.io, which allowed me to donate >$100k to EA causes, including >$60k to MIRI.
[Less important: I've also started a project to translate 80,000 Hours, a career guide that helps to find a fulfilling career that does good, into Russian. The impact and the effectiveness aside, for a year, I was the head of the Russian Pastafarian Church: a movement claiming to be a parody religion, with 200 000 members in Russia at the time, trying to increase separation between religious organisations and the state. I was a political activist and a human rights advocate. I studied relevant Russian and international law and wrote appeals that won cases against the Russian government in courts; I was able to protect people from unlawful police action. I co-founded the Moscow branch of the "Vesna" democratic movement, coordinated election observers in a Moscow district, wrote dissenting opinions for members of electoral commissions, helped Navalny's Anti-Corruption Foundation, helped Telegram with internet censorship circumvention, and participated in and organized protests and campaigns. The large-scale goal was to build a civil society and turn Russia into a democracy through nonviolent resistance. This goal wasn't achieved, but some of the more local campaigns were successful. That felt important and was also mostly fun- except for being detained by the police. I think it's likely the Russian authorities would imprison me if I ever visit Russia.]
People do in fact try to be very goal-directed about protesting! They have a lot of institutional knowledge on it!
You can study what worked and what didn’t work in the past, and what makes a difference between a movement that succeeds and a movement that doesn’t. You can see how movements organize, how they grow local leaders, how they come up with ideas that would mobilize people.
A group doesn’t have to attempt a hunger strike to figure out what the consequences would be; it can study and think, and I expect that to be a much more valuable use of time than doing hunger strikes.
Yep, I basically believe this.
(Social movements (and comms and politics) are not easy to reason about well from first principles. I think Michael is wrong to be making this particular self-sacrifice, not because he hasn’t thought carefully about AI but because he hasn’t thought carefully about hunger strikes.)
a hunger strike will eventually kill you even if you take vitamins, electrolytes, and sugar. (a way to prevent death despite the target not giving in is often a group of supporters publicly begging the person on the hunger strike to stop and not kill themselves for some plausible reasons, but sometimes people ignore that and die.) I'm not entirely sure what Guido's intention is if Anthropic doesn't give in.
To clarify, "think for five minutes" was an appeal to people who might want to do these kinds of things in the future, not a claim about Guido or Michael.
That said, I do in fact claim they have not thought carefully about their theory of change, and the linked comment from Michael lists very obvious surface-level reasons for why do this in front of anthropic and not openai; I really would not consider this on the level of demonstrating having thought carefully about the theory of change.
At the moment, these hunger strikes are people vibe-protesting.
To clarify, I meant that the choice of actions was based on the vibes, not on careful consideration, this seeming like the right thing to do in these circuimstances.
You repeat a recommendation not to risk your life
I maybe formulated this badly.
I do not disagree with that part of your comment. I did, in fact, risk being prosecuted unjustly by the state and spending a great deal of my life in prison. I was also aware of the kinds of situations I'd want to go for hunger strikes in while in prison, though didn't think about that often.
And I, too, am willing to die to reduce the risk by a pretty small chance.
Most of the time, though, I think people who think they have this choice don't actually face it; I think the bar for risking one's life should be very high. In particular, when people have time to carefully do the math, I really want them to carefully do the math before deciding to risk their lives, and in this particular case, some of my frustration is from the people getting their math wrong.
I think as a community, we also would really want to make people err on the side of safety, and have a strong norm of assumption that most people who decide to sacrifice their lives got their math wrong. People really shouldn't be risking their lives without having carefully thought of the theory of change when they have the ability to do so.
Like, I'd bet if we find people competent in how movements achieve their goals, they will say that these particular hunger strikes are not great; and I expect it to be the case most of the time when individuals who share values with a larger movement decide to go on a hunger strike even as the larger movement thinks that would not be effective.
I think there's a very reasonable theory of change - X-risk from AI needs to enter the Overton window
While in principle, as I mentioned, a hunger strike can bring attention, this is not an effective way to do this for the particular issue that AI will kill everyone by default. The diff to communicate isn't "someone is really scared of AI ending the world"; it's "scientists think AI might literally kill everyone and also here are the reasons why".
claiming they did not think for 5 minutes
This was not a claim about these people but an appeal to potential future people to maybe do research on this stuff before making decisions like this one.
That said, I talked to Guido prior to the start of the hunger strike, tried to understand his logic, and was not convinced he had any kind of reasonable theory of change guiding his actions, and my understanding is that he perceives it as the proper action to take, in a situation like that, which is why I called this vibe-protesting.
I don't get the impression that they are
(It's not very clear what would be the conditions for them to stop the hunger strikes.)
But why not this too?
Hunger strikes can be very effective and powerful if executed wisely. My comment expresses my strong opinion that this did not happen here, not that it can't happen in general.
To be very clear, I expect large social movements that use protests as one of its forms of action to have the potential to be very successful and impactful if done well. Hunger strikes are significantly different from protests. Hunger strikes can be powerful, but they're best for very different contexts.
I have great empathy and deep respect for the courage of the people currently on hunger strikes to stop the AI race. Yet, I wish they hadn’t started them: these hunger strikes will not work.
Hunger strikes can be incredibly powerful when there’s a just demand, a target who would either give in to the demand or be seen as a villain for not doing so, a wise strategy, and a group of supporters.
I don’t think these hunger strikes pass the bar. Their political demands are not what AI companies would realistically give in to because of a hunger strike by a small number of outsiders.
A hunger strike can bring attention to how seriously you perceive an issue. If you know how to make it go viral, that is; in the US, hunger strikes are rarely widely covered by the media. And even then, you are more likely to marginalize your views than to make them go more mainstream: if people don’t currently think halting frontier general AI development requires hunger strikes, a hunger strike won’t explain to them why your views are correct: this is not self-evident just from the description of the hunger strike, and so the hunger strike is not the right approach here and now.
Also, our movement does not need martyrs. You can be a lot more helpful if you eat well, sleep well, and are able to think well and hard. Your life is also very valuable, it is a part of what we’re fighting for; saving a world without you is slightly sadder than saving a world with you; and perhaps more importantly to you, it will not help. It needs to already be seen by the public as legitimate, to make them more sympathetic towards your cause and exert pressure. It needs to target decision makers who have the means to give in and advance your cause by doing that, for it to have any meaning at all.
At the moment, these hunger strikes are people vibe-protesting. They feel like some awful people are going to kill everyone, they feel powerless, and so they find a way to do something that they perceive as having a chance of changing the situation.
Please don’t risk your life; especially, please don’t risk your life in this particular way that won’t change anything.
Action is better than inaction; but please stop and think of your theory of change for more than five minutes, if you’re planning to risk your life, and then don’t risk your life[1]; please pick actions thoughtfully and wisely and not because of the vibes[2].
You can do much more if you’re alive and well and use your brain.
Not to say that you shouldn’t be allowed to risk your life for a large positive impact. I would sacrifice my life for some small chance of preventing AI risk. But most people who think they’re facing a choice to sacrifice their life for some chance of making a positive impact are wrong and don't actually face it; so I think the bar for risking one's life should be very high. In particular, when people have time to carefully do the math, I really want them to carefully do the math before deciding to risk their lives, and in this specific case, some of my frustration is from the people clearly getting their math wrong.
I think as a community, we also would really want to make people err on the side of safety, and have a strong norm of assumption that most people who decide to sacrifice their lives got their math wrong, especially if a community that shares their values disagrees with them on the consequences of the sacrifice. People really shouldn't be risking their lives without having carefully thought of the theory of change (when they have the ability to do so).
I'd bet if we find people competent in how movements achieve their goals, they will say that these particular hunger strikes are not great; and I expect it to be the case most of the time when individuals who share values with a larger movement decide to go on a hunger strike even as the larger movement thinks that would not be effective.
My strong impression is that the person on the hunger strike in front of Anthropic is doing this primarily because he feels like it is the proper thing to do in this situation, like it’s the action someone should be taking here.
I think it's a bad sign that Anthropic seemingly actively sought out an article that ended up being wrong/misleading in a way which was convenient for Anthropic at the time and then didn't correct it.
Yep; misleading the public about this doesn’t exactly boost confidence in how much Anthropic would prioritize integrity/commitments/etc. when their interests are on the line.
(I agree with Lucious in that I think it is important that people have the option of getting cryopreserved and also are aware of all the reality-fluid stuff before they decide to kill themselves.)