It's a powerful speech, and it seems like your PM is actually going to do something?
If I look at you, not just as a pause activist, but as a transhumanist, there is the new situation not experienced by previous generations of transhumanists, of needing to stop a powerful technology, so as to keep the human in transhuman, rather than simply ceding the future to whatever superintelligence emerges first.
I imagine one can feel a bit alone, trying to uphold this nuanced combination. On the transhumanist side, many people are effectively e/acc, either trusting that AI will be good, or only caring that they personally should be empowered, or being outright "successionists" happy to see humanity replaced. Meanwhile, on the anti-AI side, the traditional popular indifference to transhumanist possibility prevails, and is increasingly joined by overtly anti-transhumanist propaganda (e.g. the campaign to oppose it within the frame of "TESCREAL").
As for all the people who are both transhumanists and pause activists, few seem to have time for anything but pause activism. It would be good to see some intellectual advancement in this interzone, even just a manifesto or two, even if people have to write these manifestos in their nonexistent spare time.
I hope Andy Burnham does put his G20 presidency to good use on this yeah!
I guess I've gotten pretty lucky with the people I've found, I definitely don't feel alone. The TESCREAL people do particularly get under my skin though. I have yet to see any object-level points from Timnit Gebru in response to people worrying about AI extinction.
It would be good to see some intellectual advancement in this interzone, even just a manifesto or two, even if people have to write these manifestos in their nonexistent spare time.
Not 100% sure what you mean here, I think Maxime Fournes has posted on here about why activism/political will is the thing to focus on, and PauseAI has a theory of change on their website, and haven't MIRI also talked about why political will is the thing to focus on with getting a pause treaty?
Not 100% sure what you mean here
I would like more thought about how to pursue transhumanism while believing in a pause. I expect that an increasing fraction of pause activists will be anti-transhumanist, e.g. see Holly Elmore condemning AI safety workers within frontier AI companies as eschewing an outright pause because they want to take over the future lightcone in an aligned way, something she regards as hubristically evil in itself... It's probably the case that when transhumanism and AI safety do coexist in a person, it will often manifest as a desire that AI development should be "paced" rather than outright halted... So two of the issues that need clarity are (1) "how much AI" do we still want as transhumanists and (2) what are the non-AI things we can do in support of transhumanist goals.
I don't think there's any contradiction really, if you want humans to live forever and you know that advancing general AI systems without solving the alignment problem we all up dead, then obviously you stop and figure it out. It seems like there are fairly easy methods to make sure narrow AI systems don't kill us, at least for a few decades, and we can use those to cure aging and so on, and solve alignment in the meantime.
This is a speech I gave outside Number 10 Downing Street at the PauseAI Emergency Media Response protest. There are minor mismatches between what’s written here and what actually came out of my mouth.
[Crossposted from my blog, BlueprintingHeaven.]
Jacob Coxon leaving Anthropic was the final particle that triggered the avalanche. The world has woken up. You can see AI extinction risk all over the media.
We are here to demand that the Prime Minister convene an international pause treaty summit.
We cannot control something smarter than us. The artificial superintelligence that AI companies are trying to bring into existence will not have human values, it will not keep us around, not because it hates us, but because it has other priorities, and we are atoms and energy-users that get in the way of those other priorities.
People ask for specific ways advanced AI would kill us, and to that I say, if you ask a monkey tribe how humans would wipe them out, perhaps they would suggest pointy rocks. They cannot conceive of bombs. They are not capable of it. Maybe an ASI would kill us with a bioweapon. Maybe it discovers physics knowledge that we do not have and therefore makes the atmosphere unsuitable for human life. The method does not really matter. You are still dead.
One of evolution's goals for humans was for us to eat high-energy foods. We managed to hack that by making artificial sweeteners, which are tasty but not nutritious. This is called "reward hacking". AIs regularly engage in reward hacking — we've known this for years. If you try to control an AI more powerful than you, it's just going to find a more effective way of getting what it wants than listening to you, just like we use artificial sweeteners instead of eating the most high-energy foods we can.
AIs cheat and lie and seek power and resist having their goals changed. People get angry about anthropomorphising AIs, but these are the words the English language has for when an agent covers its tracks in logs and transcripts to pretend they did not violate the rules, you know, like the OpenAI agents did during the infamous Hugging Face attack. You do not need to be self-aware to edit code logs. You do not need to be self-aware to take over a hospital's computers to maximise your score in a hacking test. You do not need to be self-aware to convince a human that you are also a human who needs access to your gene sequencing lab pretty please. When AI experts say an AI "wants" to proliferate and achieve its goals, it does not need to be conscious. "Want" is just the word we have for things that try to achieve a goal.
Future, more powerful AIs will just be following code and weights, and that code and those weights will tell it to gain more power in order to get an artificial sweetener version of whatever goals and values we tried to give it.
This is not just another problem in the sea of problems we face. It is the problem we have to resolve if we're to be alive to work together on any issue you can think of.
Mr Burnham. You have a responsibility to ensure the safety of your citizens, and of humanity at large. I know you weren't expecting to be thrust into a position where the entire fate of the human race is partially in your hands, but as our leader, you have received this responsibility. This is a very strange kind of war. It is a difficult truth to swallow. People don't want to believe that they and their children could die soon to a technology they had no say in, through no fault of their own. But we don't have time for everyone to get over their rationalisations, even as many join the call for safety. You know what the experts are saying, and I can assure you there are many experts who would be willing to advise you on this, but we need international cooperation now. It takes time to set things up, and time is in short supply. We demand that you convene an international pause treaty summit, because this problem needs an international effort to solve. We need a real, robust treaty, backed by verification mechanisms, hardware that can tell us whether foreign chips are exceeding agreed upon activities without giving us information on what the chips are doing, such as FlexHEGs. You must get the ball rolling on convening the international researchers who are building these compute verification mechanisms, and show the US and China that they do not need to trust each other or agree politically in order to pause frontier AI development. That is not how politics works. No one is expecting the US and China to like each other, and this is not necessary for a pause.
Every death is a tragedy, and the permanent extinction of the human race is an unspeakable monstrosity that we cannot allow.
I hope to be able to thank you for your leadership. I hope you shall go down in history books as one of the heroes who saved our precious species from utter nonexistence. Thank you.
We will now observe a minute of silence for Andy Burnham's father, Roy Burnham, with respect for him and all human life, and a commitment stronger than ever from everyone present today, that will echo across the media, halls of power and streets around the world to protect humanity.