TL;DR—We’re distributing $20k in total as prizes for submissions that make effective arguments for the importance of AI safety. The goal is to generate short-form content for outreach to policymakers, management at tech companies, and ML researchers. This competition will be followed by another competition in around a month that focuses on long-form content.
This competition is for short-form arguments for the importance of AI safety. For the competition for distillations of posts, papers, and research agendas, see the Distillation Contest.
Objectives of the arguments
To mitigate AI risk, it’s essential that we convince relevant stakeholders sooner rather than later. To this end, we are initiating a pair of competitions to build effective arguments for a range of audiences. In particular, our audiences include policymakers, tech executives, and ML researchers.
- Policymakers may be unfamiliar with the latest advances in machine learning, and may not have the technical background necessary to understand some/most of the details. Instead, they may focus on societal implications of AI as well as which policies are useful.
- Tech executives are likely aware of the latest technology, but lack a mechanistic understanding. They may come from technical backgrounds and are likely highly educated. They will likely be reading with an eye towards how these arguments concretely affect which projects they fund and who they hire.
- Machine learning researchers can be assumed to have high familiarity with the state of the art in deep learning. They may have previously encountered talk of x-risk but were not compelled to act. They may want to know how the arguments could affect what they should be researching.
We’d like arguments to be written for at least one of the three audiences listed above. Some arguments could speak to multiple audiences, but we expect that trying to speak to all at once could be difficult. After the competition ends, we will test arguments with each audience and collect feedback. We’ll also compile top submissions into a public repository for the benefit of the x-risk community.
Note that we are not interested in arguments for very specific technical strategies towards safety. We are simply looking for sound arguments that AI risk is real and important.
Competition details
The present competition addresses shorter arguments (paragraphs and one-liners) with a total prize pool of $20K. The prizes will be split among, roughly, 20-40 winning submissions. Please feel free to make numerous submissions and try your hand at motivating various different risk factors; it's possible that an individual with multiple great submissions could win a good fraction of the prize. The prize distribution will be determined by effectiveness and epistemic soundness as judged by us. Arguments must not be misleading.
To submit an entry:
- Please leave a comment on this post (or submit a response to this form), including:
- The original source, if not original.
- If the entry contains factual claims, a source for the factual claims.
- The intended audience(s) (one or more of the audiences listed above).
- In addition, feel free to adapt another user’s comment by leaving a reply—prizes will be awarded based on the significance and novelty of the adaptation.
Note that if two entries are extremely similar, we will, by default, give credit to the entry which was posted earlier. Please do not submit multiple entries in one comment; if you want to submit multiple entries, make multiple comments.
The first competition will run until May 27th, 11:59 PT. In around a month, we’ll release a second competition for generating longer “AI risk executive summaries'' (more details to come). If you win an award, we will contact you via your forum account or email.
Paragraphs
We are soliciting argumentative paragraphs (of any length) that build intuitive and compelling explanations of AI existential risk.
- Paragraphs could cover various hazards and failure modes, such as weaponized AI, loss of autonomy and enfeeblement, objective misspecification, value lock-in, emergent goals, power-seeking AI, and so on.
- Paragraphs could make points about the philosophical or moral nature of x-risk.
- Paragraphs could be counterarguments to common misconceptions.
- Paragraphs could use analogies, imagery, or inductive examples.
- Paragraphs could contain quotes from intellectuals: “If we continue to accumulate only power and not wisdom, we will surely destroy ourselves” (Carl Sagan), etc.
For a collection of existing paragraphs that submissions should try to do better than, see here.
Paragraphs need not be wholly original. If a paragraph was written by or adapted from somebody else, you must cite the original source. We may provide a prize to the original author as well as the person who brought it to our attention.
One-liners
Effective one-liners are statements (25 words or fewer) that make memorable, “resounding” points about safety. Here are some (unrefined) examples just to give an idea:
- Vladimir Putin said that whoever leads in AI development will become “the ruler of the world.” (source for quote)
- Inventing machines that are smarter than us is playing with fire.
- Intelligence is power: we have total control of the fate of gorillas, not because we are stronger but because we are smarter. (based on Russell)
One-liners need not be full sentences; they might be evocative phrases or slogans. As with paragraphs, they can be arguments about the nature of x-risk or counterarguments to misconceptions. They do not need to be novel as long as you cite the original source.
Conditions of the prizes
If you accept a prize, you consent to the addition of your submission to the public domain. We expect that top paragraphs and one-liners will be collected into executive summaries in the future. After some experimentation with target audiences, the arguments will be used for various outreach projects.
(We thank the Future Fund regrant program and Yo Shavit and Mantas Mazeika for earlier discussions.)
In short, make a submission by leaving a comment with a paragraph or one-liner. Feel free to enter multiple submissions. In around a month we'll divide 20K to award the best submissions.
I'd like to complain that this project sounds epistemically absolutely awful. It's offering money for arguments explicitly optimized to be convincing (rather than true), it offers money only for prizes making one particular side of the case (i.e. no money for arguments that AI risk is no big deal), and to top it off it's explicitly asking for one-liners.
I understand that it is plausibly worth doing regardless, but man, it feels so wrong having this on LessWrong.
If the world is literally ending, and political persuasion seems on the critical path to preventing that, and rationality-based political persuasion has thus far failed while the empirical track record of persuasion for its own sake is far superior, and most of the people most familiar with articulating AI risk arguments are on LW/AF, is it not the rational thing to do to post this here?
I understand wanting to uphold community norms, but this strikes me as in a separate category from “posts on the details of AI risk”. I don’t see why this can’t also be permitted.
TBC, I'm not saying the contest shouldn't be posted here. When something with downsides is nonetheless worthwhile, complaining about it but then going ahead with it is often the right response - we want there to be enough mild stigma against this sort of thing that people don't do it lightly, but we still want people to do it if it's really clearly worthwhile. Thus my kvetching.
(In this case, I'm not sure it is worthwhile, compared to some not-too-much-harder alternative. Specifically, it's plausible to me that the framing of this contest could be changed to not have such terrible epistemics while still preserving the core value - i.e. make it about fast, memorable communication rather than persuasion. But I'm definitely not close to 100% sure that would capture most of the value.
Fortunately, the general policy of imposing a complaint-tax on really bad epistemics does not require me to accurately judge the overall value of the proposal.)
No, it's just the standard frontpage policy:
Technically the contest is asking for attempts to persuade not explain, rather than itself attempting to persuade not explain, but the principle obviously applies.
As with my own comment, I don't think keeping the post off the frontpage is meant to be a judgement that the contest is net-negative in value; it may still be very net positive. It makes sense to have standard rules which create downsides for bad epistemics, and if some bad epistemics are worthwhile anyway, then people can pay the price of those downsides and move forward.
Raemon and I discussed whether it should be frontpage this morning. Prizes are kind of an edge case in my mind. They don't properly fulfill the frontpage criteria but also it feels like they deserve visibility in a way that posts on niche topics don't, so we've more than once made an exception for them.
I didn't think too hard about the epistemics of the post when I made the decision to frontpage, but after John pointed out the suss epistemics, I'm inclined to agree, and concurred with Raemon moving it back to Personal.
----
I think the prize could be improved simply by rewarding the best arguments in favor and against AI risk. This might actually be more convincing to the skeptics – we paid people to argue against this position and now you can see the best they came up with.
We're out of time. This is what serious political activism involves.
Most movements (and yes, this is a movement) have multiple groups of people, perhaps with degrees in subjects like communication, working full time coming up with slogans, making judgments about which terms to use for best persuasiveness, and selling the cause to the public. It is unusual for it to be done out in the open, yes. But this is what movements do when they have already decided what they believe and now have policy goals they know they want to achieve. It’s only natural.
You didn't refute his argument at all, you just said that other movements do the same thing. Isn't the entire point of rationality that we're meant to be truth-focused, and winning-focused, in ways that don't manipulate others? Are we not meant to hold ourselves to the standard of "Aim to explain, not persuade"? Just because others in the reference class of "movements" do something doesn't mean it's immediately something we should replicate! Is that not the obvious, immediate response? Your comment proves too much; it could be used to argue for literally any popular behavior of movements, including canceling/exiling dissidents.
Do I think that this specific contest is non-trivially harmful at the margin? Probably not. I am, however, worried about the general attitude behind some of this type of recruitment, and the justifications used to defend it. I become really fucking worried when someone raises an entirely valid objection, and is met with "It's only natural; most other movements do this".
That idea seems reasonable at first glance, but upon reflection, I think it's a really bad idea. It's one thing to run a red-teaming competition, it's another to spend money building rhetorically optimised tools for the other side. If we do that, then maybe there was no point running the competition in the first place as it might all cancel out.
Wait, the goal here, at least, isn't to produce truth, it is to disseminate it. Counter-arguments are great, but this isn't about debating the question, it's about communicating a conclusion well.
It's unreasonable to look at AI and think "in order to make it more predictable, more controllable, we just have to make it smarter". When individual humans become smarter, they become much less predictable. And the human mind is built to predict the behavior of other humans.
Random side note: GPT-3 seems to be able to generate decent one liners.
Generate one-liners describing the dangers of AI. An example is "Inventing machines that are smarter than us is playing with fire."
1. Machines that are smarter than us could easily become our masters.
2. If we're not careful, AI could spell the end of humanity as we know it.
3. AI could be used to create weapons of mass destruction that could devastate the planet.
4. AI could be used to create powerful robots that could enslave humans.
5. AI could be used to create artificial intelligence that could turn against its creators
6. AI usher in a new era of cyber-warfare that could cripple society
7. AI could create self-replicating robots that could eventually consume all resources on Earth
8. AI could be used to create powerful AIs that could eventually surpass human intelligence and take over the world
9. AI technology could eventually be used to create a global surveillance state where everyone is constantly watched and monitored
Any arguments for AI safety should be accompanied by images from DALL-E 2.
One of the key factors which makes AI safety such a low priority topic is a complete lack of urgency. Dangerous AI seems like a science fiction element, that's always a century away, and we can fight against this perception by demonstrating the potential and growth of AI capability.
No demonstration of AI capability has the same immediate visceral power as DALL-E 2.
In longer-form arguments, urgency could also be demonstrated through GPT-3's prompts, but DALL-E 2 is better, especially ... (read more)
I remember watching a documentary made during the satanic panic by some activist Christian group. I found it very funny at the time, and then became intrigued when an expert came on to say something like:
I was impressed with that line's simplicity and effectiveness. A lot of it's effectiveness stems silently from the fact that, inadvertently, it helps suspend disbelief about the negative impact of "s... (read more)
First two paragraphs of https://astralcodexten.substack.com/p/book-review-a-clinical-introduction seem to fit the bill.
-- Stuart Russell on a February 25, 2021 podcast with the Future of Life Institu... (read more)
Neither us humans, nor the flower, sees anything that looks like a bee. But when a bee looks at it, it sees another bee, and it is tricked into pollinating that flower. The flower did not know any of this, it's petals randomly changed shape over millions of years, and eventually one of those random shapes started tricking bees and outperforming all of the other flowers.
Today's AI already does this. If AI begins to approach human intelligence, there's no limit to the number of ways things can go horribly wrong.
Here is the spreadsheet containing the results from the competition.
More quotes on AI safety here.
[Policy makers]
A couple of years ago there was an AI trained to beat Tetris. Artificial intelligences are very good at learning video games, so it didn't take long for it to master the game. Soon it was playing so quickly that the game was speeding up to the point it was impossible to win and blocks were slowly stacking up, but before it could be forced to place the last piece, it paused the game.
As long as the game didn't continue, it could never lose.
When we ask AI to do something, like play Tetris, we have a lot of assumptions about how it can or ... (read more)
(a)
Look, we already have superhuman intelligences. We call them corporations and while they put out a lot of good stuff, we're not wild about the effects they have on the world. We tell corporations 'hey do what human shareholders want' and the monkey's paw curls and this is what we get.
Anyway yeah that but a thousand times faster, that's what I'm nervous about.
(b)
Look, we already have superhuman intelligences. We call them governments and while they put out a lot of good stuff, we're not wild about the effects they have on the world. We tell gov... (read more)
"Most AI reserch focus on building machines that do what we say. Aligment reserch is about building machines that do what we want."
Source: Me, probably heavely inspred by "Human Compatible" and that type of arguments. I used this argument in conversations to explain AI Alignment for a while, and I don't remember when I started. But the argument is very CIRL (cooperative inverse reinforcment learning).
I'm not sure if this works as a one liner explanation. But it does work as a conversation starter of why trying to speify goals directly is a bad idea. And ho... (read more)
Crypto Executives and Crypto Researchers
(Might resonate well with crypto folks.)
"Humanity has risen to a position where we control the rest of the world precisely because of our [unrivaled] mental abilities. If we pass this mantle to our machines, it will be they who are in this unique position."
Toby Ord, The Precipice
Replaced [unparalleled] with [unrivaled]
As recent experience has shown, exponential processes don't need to be smarter than us to utterly upend our way of life. They can go from a few problems here and there to swamping all other considerations in a span of time too fast to react to, if preparations aren't made and those knowledgeable don't have the leeway to act. We are in the early stages of an exponential increase in the power of AI algorithms over human life, and people who work directly on these problems are sounding the alarm right now. It is plausible that we will soon have processes that... (read more)
For policymakers:
Expecting today's ML researchers to understand AGI is like expecting a local mechanic to understand how to design a more efficient engine. It's a lot better than total ignorance, but it's also clearly not enough.
In 1903, The New York Times thought heavier-than-air flight would take 1-10 million years… less than 10 weeks before it happened. Is AI next? (source for NYT info) (Policymakers)
If AI approaches and reaches human-level intelligence, it will probably pass that level just as quickly as it arrived at that level.
What about graphics? e.g. https://twitter.com/DavidSKrueger/status/1520782213175992320
"AI will probably surpass human intelligence at the same pace that it reaches human intelligence. Considering the pace of AI advancement over the last 3 years, that pace will probably be very fast"
https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-
“The smartest ones are the most criminally capable.” [·]
"AI cheats. We've seen hundreds of unique instances of this. It finds loopholes and exploits them, just like us, only faster. The scary thing is that, every year now, AI becomes more aware of its surroundings, behaving less like a computer program and more like a human that thinks but does not feel"
Imagine (an organisation like) the catholic church, but immortal, never changing, highly competent and relentlessly focused on its goals - it could control the fate of humanity for millions of years.
(Policymakers) There is outrage right now about AI systems amplifying discrimination and polarizing discourse. Consider that this was discovered after they were widely deployed. We still don't know how to make them fair. This isn't even much of a priority.
Those are the visible, current failures. Given current trajectories and lack of foresight of AI research, more severe failures will happen in more critical situations, without us knowing how to prevent them. With better priorities, this need not happen.
On average, experts estimate a 10-20% (?) probability of human extinction due to unaligned AGI this century, making AI Safety not simply the most important issue for future generations, but for present generations as well. (policymakers)
Clarke’s First Law goes: When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
Stuart Russell is only 60. But what he lacks in age, he makes up in distinction: he’s a computer science professor at Berkeley, neurosurgery professor at UCSF, DARPA advisor, and author of the leading textbook on AI. His book Human Compatible states that superintelligent AI is possible; Clarke would recommend we listen.
(tech executives, ML researchers)
(ada... (read more)
All humans, even people labelled "stupid", are smarter than apes. Both apes and humans are far smarter than ants. The intelligence spectrum could extend much higher, e.g. up to a smart AI… (Adapted from here). (Policymakers)
The smarter AI gets, the further it strays from our intuitions of how it should act. (Tech executives)
The existence of the human race has been defined by our absolute monopoly on intelligent thought. That monopoly is no longer absolute, and we are on track for it to vanish entirely.
Here's my submission, it might work better as bullet points on a page.
AI will transform human societies over the next 10-20 years. Its impact will be comparable to electricity or nuclear weapons. As electricity did, AI could improve the world dramatically; or, like nuclear weapons, it could end it forever. Like inequality, climate change, nuclear weapons, or engineered pandemics, AI Existential Risk is a wicked problem. It calls upon every policymaker to become a statesperson: to rise above the short-term, narrow inte... (read more)
(To Policymakers and Machine Learning Researchers)
Building a nuclear weapon is hard. Even if one manages to steal the government's top secret plans, one still need to find a way to get uranium out of the ground, find a way to enrich it, and attach it to a missile. On the other hand, building an AI is easy. With scientific papers and open source tools, researchers are doing their utmost to disseminate their work.
It's pretty hard to hide a uranium mine. Downloading TensorFlow takes one line of code. As AI becomes more powerful and more dangerous, greater efforts need to be taken to ensure malicious actors don't blow up the world.
To Policymakers: "Just think of the way in which we humans have acted towards animals, and how animals act towards lesser animals, now think of how a powerful AI with superior intellect might act towards us, unless we create them in such a way that they will treat us well, and even help us."
Source: Me
Target: Everyone. Another good zinger.
Source: https://nitter.nl/liminal_warmth/status/1511536700127731713#m
You wouldn't hire an employee without references. Why would you make an AI that doesn't share your values?
(policymakers, tech executives)
For policymakers:
The predictability of today's AI systems doesn't tell us squat about whether they will remain predictable after achieving human-level intelligence. Individual apes are far more predictable than individual humans, and apes themselves are far less predictable than ants.
Climate change was weird in the 1980s. Pandemics were weird in the 2010s. Every world problem is weird... until it happens.
(policymakers)
When nuclear weapons were first made, there was a serious concern that the first nuclear test would trigger a chain reaction and ignite the entire plant’s atmosphere. AI has an analogous issue. It used a technology called machine learning, that allows AI to figure out the solutions for problems on its own. The problem is that we don’t know whether this technology, or something similar, might cause the AI to start “thinking for itself.” There are a significant number of software engineers who think this might have disastrous consequences, but it’s a risk to... (read more)
Imagine it's 1932, but with one major difference: uranium is cheap enough that anyone can get some. Radioactive materials are unregulated. The world's largest companies are competing to build nuclear power plants. Nuclear weapons have not yet been discovered. Would you think nuclear arms control is premature? Or would you want to get started now to prevent a catastrophe?
This is the same situation the real world is in, with machine learning and artificial intelligence. The world's biggest tech companies are gathering GPUs and working to build AI that is smarter than humans about everything. And right now, there's not much coordination being done to make this go well. (Policymakers)
“Thousands of researchers at the worlds richest corporations are all working to make AI more powerful. Who is working to make AI more moral?”
(For policymakers and activists skeptical of big tech)
If an AI is cranked up to the point that it becomes smarter than humans, it will not behave predictably. We humans are not predictable. Even chimpanzees and dolphins are unpredictable. Smart things are not predictable. Intelligence, itself, does not tend to result in predictability.
"When I visualize [a scenario where a highly intelligent AI compromises all human controllers], I think it [probably] involves an AGI system which has the ability to be cranked up by adding more computing resources to it [to increase its intelligence and creativity incrementally]; and I think there is an extended period where the system is not aligned enough that you can crank it up that far, without [any dangerously erratic behavior from the system]"
Eliezer Yudkowsky
"Guns were developed centuries before bulletproof vests"
"Smallpox was used as a tool of war before the development of smallpox vaccines"
EY, AI as a pos neg factor, 2006ish
Targeting policymakers:
The implicit goal, the thing you want, is to get good at the game; the explicit goal, the thing the AI was programmed to want, is to rack up points by any means necessary. (Machine learning researchers)
[Policy makers & ML researchers]
"There isn’t any spark of compassion that automatically imbues computers with respect for other sentients once they cross a certain capability threshold. If you want compassion, you have to program it in" (Nate Soares). Given that we can't agree on whether a straw has two holes or one...We should probably start thinking about how program compassion into a computer.
Remember all the scary stuff the engineers said a terrorist could think to do? Someone could write a computer program to do them just randomly.
It is a fundamental law of thought that thinking things will cut corners, misinterpret instructions, and cheat.
"Past technological revolutions usually did not telegraph themselves to people alive at the time, whatever was said afterward in hindsight"
Eliezer Yudkowsky, AI as a pos neg factor, around 2006
"Imagine that Facebook and Netflix have two separate AIs that compete over hours that each user spends on their own platform. They want users to spend the maximum amount of minutes on Facebook or Netflix, respectively.
The Facebook AI discovers that posts that spoil popular TV shows result in people spending more time on the platform. It doesn't know what spoilers are, only that they cause people to spend more time on Facebook. But in reality, they're ruining the entertainment value from excellent shows on Netflix.
Even worse, the Netflix AI discovers ... (read more)
[Policymakers & ML researchers]
A virus doesn't need to explain itself before it destroys us. Neither does AI.
A meteor doesn't need to warn us before it destroys us. Neither does AI.
An atomic bomb doesn't need to understand us in order to destroy us. Neither does AI.
A supervolcano doesn't need to think like us in order to destroy us. Neither does AI.
(I could imagine a series riffing based on this structure / theme)
[ML researchers]
Given that we can't agree on whether a hotdog is a sandwich or not...We should probably start thinking about how to tell a computer what is right and wrong.
[Insert call to action on support / funding for AI governance / regulation etc.]
-
Given that we can't agree on whether a straw has two holes or one...We should probably start thinking about how to explain good and evil to a computer.
[Insert call to action on support / funding for AI governance / regulation etc.]
(I could imagine a series riffing based on this structure / theme)
I will post my submissions as individual replies to this comment. Please let me know if there’s any issues with that.
Question: "effective arguments for the importance of AI safety" - is this about arguments for the importance of just technical AI safety, or more general AI safety, to include governance and similar things?
“Aligning AI is the last job we need to do. Let’s make sure we do it right.”
(I’m not sure which target audience my submissions are best targeted towards. I’m hoping that the judges can make that call for me.)
I recently talked with the minister of innovation in Yucatan, and ze's looking to have competitions in the domain of artificial intelligence in a large conference on innovation they're organizing in Yucatan, Mexico that will happen in mid-November. Do you think there's the potential for a partnership?
Have any prizes been awarded yet? I haven't heard anything about prizes, but that could have just been that I didn't win one...
AI existential risk is like climate change. It's easy to come up with short slogans that make it seem ridiculous. Yet, when you dig deeper into each counterargument, you find none of them are very convincing, and the dangers are quite substantial. There's quite a lot of historical evidence for the risk, especially in the impact humans have had on the rest of the world. I strongly encourage further, open-minded study.
Policymakers
This is a an extremely common, perhaps even overused, catchphrase for AI risk. But I still want to make sure it’s represented because I personally find it striking.
Leading up to the first nuclear weapons test, the Trinity event in July 1945, multiple physicists in the Manhattan Project thought the single explosion would destroy the world. Edward Teller, Arthur Compton, and J. Robert Oppenheimer all had concerns that the nuclear chain reaction could ignite Earth's atmosphere in an instant. Yet, despite disagreement and uncertainty over their calculations, they detonated the device anyway. If the world's experts in a field can be uncertain about causing human extinction with their work, and still continue doing it, wha... (read more)
Target: Everyone? Just really snappy.
Source: https://nitter.nl/Miles_Brundage/status/1490512439154151426#m
Machine learning researchers
Common Deep Learning Critique “It’s just memorization”
Critique:
... (read more)Policymakers
These researchers built an AI for discovering less toxic drug compounds. Then they retrained it to do the opposite. Within six hours it generated 40,000 toxic molecules, including VX nerve agent and "many other known chemical warfare agents."
Source: https://nitter.nl/WriteArthur/status/1503393942016086025#m
Imagine a piece of AI software was invented, capable of doing any intellectual task a human can, at a normal human level. Should we be concerned about this? Yes, because this artificial mind would be more powerful (and dangerous) than any human mind. It can think anything a normal human can, but faster, more precisely, and without needing to be fed. In addition, it could be copied onto a million computers with ease. An army of thinkers, available at the press of a button. (Adapted from here). (Policymakers)
Policymakers and techxecutives:
If we build an AI that's smarter than a human, then it will be smarter than a human, so it won't have a hard time convincing us that it's on our side. This is why we have to build it perfectly, before it's built, not after.
AI has a history of surprising us with its capabilities. Throughout the last 50 years, AI and machine learning systems have kept gaining skills that were once thought to be uniquely human, such as playing chess, classifying images, telling stories, and making art. Already, we see the risks associated with these kinds of AI capabilities. We worry about bias in algorithms that guide sentencing decisions or polarization induced by algorithms that curate our social media feeds. But we have every reason to believe that trends in AI progress will continue. AI wi... (read more)
For lefties:
For right-wingers:
Policymakers
Look, we know how we sound waving our hands warning about this AI stuff. But here’s the thing, in this space, things that sounded crazy yesterday can become very real overnight. (Link DALL-E 2 or Imagen samples). Honestly ask yourself: would you have believed a computer could do that before seeing these examples? And if you were surprised by this, how many more surprises won’t you see coming? We’re asking you to expect to be surprised, and to get ready.
Humans are pretty clever, but AI will be eventually be even more clever. If you give a powerful enough AI a task, it can direct a level of ingenuity towards it far greater than history’s smartest scientists and inventors. But there are many cases of people accidentally giving an AI imperfect instructions.
If things go poorly, such an AI might notice that taking over the world would give it access to lots of resources helpful for accomplishing its task. If this ever happens, even once, with an AI smart enough to escape any precautions we set and succeed at taking over the world, then there will be nothing humanity can do to fix things.
The first moderately smart AI anyone develops might quickly become the last time that that people are the smartest things around. We know that people can write computer programs. Once we make an AI computer program that is a bit smarter than people, it should be able to write computer programs too, including re-writing its own software to make itself even smarter. This could happen repeatedly, with the program getting smarter and smarter. If an AI quickly re-programs itself from moderately-smart to super-smart, we could soon find that it is as disinterested in the wellbeing of people as people are of mice.
(For non-x-risk-focused transhumanists, some of whom may be tech execs or ML researchers.)
Some people treat the possibility of human extinction with a philosophical detachment: who are we to obstruct the destiny of the evolution of intelligent life? If the "natural" course of events for a biological species like ours is to be transcended by our artificial "mind children", shouldn't we be happy for them?
I actually do have some sympathy for this view, in the sense that the history where we build AI that kills us is plausibly better than the history where the... (read more)
"Through the past 4 billion years of life on earth, the evolutionary process has emerged to have one goal: create more life. In the process, it made us intelligent. In the past 50 years, as humanity gotten exponentially more economically capable we've seen human birth rates fall dramatically. Why should we expect that when we create something smarter than us, it will retain our goals any better than we have retained evolution's?" (Policymaker)
A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. - Stuart Russell
With enough effort, could humanity have prevented nuclear proliferation? (Machine learning researchers)
Progress moves faster than we think; who in the past would've thought that the world economy would double in size, multiple times in a single person's lifespan? (Adapted from here). (Policymakers)
The nightmare scenario is that we find ourselves stuck with a catalog of mature, powerful, publicly available AI techniques... which cannot be used to build Friendly AI without redoing the last three decades of AI work from scratch.
EY, AI as a pos neg factor, 2006ish
There are several dozens scenarios how advanced AI can cause a global catastrophe. The full is presented in the article Classification of Global Catastrophic Risks Connected with Artificial Intelligence. At least some scenarios are real and likely to happen. Therefore we have to pay more attention to AI safety.
Google's DeepMind has 4 pages of blog posts about their fast-moving research to build artificial intelligence that can solve problems on its own. In contrast, they have only 2 posts total about the ethics and safeguards for doing so. We can't necessarily rely on the top AI labs in the world, to think of everything that could go wrong with their increasingly-powerful systems. New forms of oversight, nimbler than government regulation or IRBs, need to be invented to keep this powerful technology aligned with human goals. (Policymakers)
Helpfully, DeepMind's chief operating officer, Lila Ibrahim ("a passionate advocate for social impact in her work and her personal life"), who would be intimately involved in any funding of safety research, overseeing large-sale deployment, and reacting to problems, has a blog post all about what she thinks AI safety is about and what she is concerned about in doing AI research responsibly: "Building a culture of pioneering responsibly: How to ensure we benefit society with the most impactful technology being developed today"
She has also written enthusiastically about DM's funding for "racial justice efforts".
A fool was tasked with designing a deity. The result was awesomely powerful but impoverished - they say it had no ideas on what to do. After much cajoling, it was taught to copy the fool’s actions. This mimicry it pursued, with all its omnipotence.
The fool was happy and grew rich.
And so things went, ‘til the land cracked, the air blackened, and azure seas became as sulfurous sepulchres.
As the end grew near, our fool ruefully mouthed something from a slim old book: ‘Thou hast made death thy vocation, in that there is nothing contemptible.’
[Tech Executives]
Look at how the world has changed in your lifetime, and how that change is accelerating. We're headed somewhere strange, and we're going to need to invest in making sure our technology remains beneficial to get a future we want.
How hard did you think about killing the last cockroach you found in your house? We're the cockroaches, and we are in the AI's house. For policy-makers, variant on the anthill argument, original source unknown
Flowers evolved to trick insects into spreading their pollen, not to feed the insects. AI also evolves; it doesn't know, it just does whatever seems to gain approval.
For policymakers
"AI keeps finding new ways to cheat"
[Policy makers & ML researchers]
“If a distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong” (Arthur Clarke). In the case of AI, the distinguished scientists are saying not just that something is possible, but that it is probable. Let's listen to them.
[Insert call to action]
The wait but why post on AI is a gold mine of one-liners and one-liner inspiration.
Part 2 has better inspiration for appealing to AI scientists.
[Policy makers & ML researchers]
“AI doesn’t have to be evil to destroy humanity – if AI has a goal and humanity just happens to come in the way, it will destroy humanity as a matter of course without even thinking about it, no hard feelings” (Elon Musk).
[Insert call to action]
It's not a question of "if" we build something smarter than us, it's a question of "when". Progress in that direction has been constant, for more than a decade now, and recently it has been faster than ever before.
"As AI gradually becomes more capable of modelling and understanding its surroundings, the risks associated with glitches and unpredictable behavior will grow. If artificial intelligence continues to expand exponentially, then these risks will grow exponentially as well, and the risks might even grow exponentially shortly after appearing"
I may have missed the deadline, but in any event:
At the rate AI is developing, we will likely develop an artificial superhuman intelligence within our lifetimes. Such a system could alter the world in ways that seem like science fiction to us, but would be trivial for it. This comes with terrible risks for the fate of humanity. The key danger is not that a rival nation or unscrupulous corporate entity will control such a system, but that no one will. As such, the system could quite possibly alter the world in ways that no human woul... (read more)
[Meta comment]
The deadline is past, should we keep the submissions coming or is it too late? Some of the best arguments I could find elsewhere are rather long, in the vein of the Superintelligence FAQ. I did not want to copy-paste chunks of it and the arguments stand better as part of a longer format.
Anyway, signalling that the lack of money incentive will not stop me from trying to generate more compelling arguments... but I'd rather do it in French instead of posting here (I'm currently working on some video scripts on AI alignment, there's not enough French content of that type).
Machine Learning Researchers
What did you think of Deep Learning in 2017? Did you predict what Deep Learning would accomplish in 2022? If not, does this change your prediction of what Deep Learning will be capable of in 2027?
If it seems bizarre to think of an entity nobody can see ruling a country, keep in mind that there is a grand tradition of dictators – most famously Stalin – who out of paranoia retreated to some secret hideaway and ruled their country through correspondence. The AI would be little different. (Directly quoted from here). (Policymakers)
It's easy to imagine that the AI will have an off switch, and that we could keep it locked in a box and ask it questions. But just think about it. If some animals were to put you in a box, do you think you would stay in there forever? Or do you think you'd figure a way out that they hadn't thought of?
AI x-risk. It sounds crazy for two reasons. One, because we are used to nothing coming close to human intelligence, and two, because we are used to AI being unintelligent. For the first, the only point of comparison is imagining something that is to us what we are to cats. For the second, though we have not quite succeeded yet, it only takes one. If you have been following the news, we are getting close.
Machine Learning Researchers
Not exactly a paraphrase or an argument, but just try to get them to plug their assumptions about AI progress into Ajeya's model.
Daniel Kokotajilo explains better more articulately than I could why this could be persuasive:
https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines?commentId=o3k4znyxFSnpXqrdL
... (read more)Safeguarding our future is not left or right, not eastern or western, not owned by the rich or the poor. It is not partisan. … Everyone has a stake in our future and we must work together to protect it. (Quoted from The Precipice by Toby Ord). (Policymakers)
Policymakers
It’s not about intent or bad actors. In a completely peaceful world with no war and no conflict, AGI would still kill everyone if we developed it and couldn't solve the alignment problem.
All but first sentence generally from: https://nitter.nl/robbensinger/status/1513994192699068419#m
Policymakers.
... (read more)How soon will smarter-than-human AI come along? While experts disagree on when this will happen, there's fear that it could happen suddenly when it does. Governments, militaries, large tech companies, and small startups are all working on superintelligent AI. Most of them see it as a competitive advantage, and would hate to reveal their progress to competing groups. Thus, the ongoing development of the project that ends up succeeding, will probably be kept secret until the last moment. (Policymakers)
Can we contain a smart AI program, simply by not giving it control of robotic appendages? Unfortunately, no. Many people have attained vast power and influence without their physical presence being required. Think of famous authors, or Stalin ruling Russia through memos, or the anonymous person who became a billionaire by inventing Bitcoin. And, if the AI is sufficiently smarter than humans, what's to stop it from hacking (or bribing or tricking) its way into critical systems? (Adapted from here). (Policymakers, Tech executives)
Machine learning researchers
Many disagreements about the probability of existential risk due to AGI involve different intuitions about what the default scenario is going to be. Some people suspect that if we don’t have an ironclad reason to suspect AGI will go well, it will almost certainly go poorly.[3] Other people think that the first thing we try has a reasonable chance of going fairly well.[4] One can imagine a spectrum with “disaster by default” on one side and “alignment by default” on the other. To the extent that one is closer to “disaster by defa... (read more)
I have a near completed position paper on the very real topic at hand. It is structured short form argument (falling under paragraph(s) && targets all 3 audiences.
I was only made aware of this "quote" call to arms w/in the last few days. I am requesting a 24 hr extension- an exception to the rule(s), I fully recognize given the parameters this request may be turned down.
Thank you for your non-AI consideration :)
AI systems are given goals by their creators—your GPS’s goal is to give you the most efficient driving directions; Watson’s goal is to answer questions accurately. And fulfilling those goals as well as possible is their motivation. One way we anthropomorphize is by assuming that as AI gets super smart, it will inherently develop the wisdom to change its original goal—but [ethicist] Nick Bostrom believes that intelligence-level and final goals are orthogonal, meaning any level of intelligence can be combined with any final goal. … Any assumption that once s... (read more)
When you hear about "the dangers of AI", what do you think of? Probably a bad actor using AI to hurt others, or a sci-fi scenario of robots turning evil. However, the bigger harm is more likely to be misalignment: an AI smarter than humans, without sharing human values. The top research labs, at places like DeepMind and OpenAI, are working to create superhuman AI, yet the current paradigm trains AI with simple goals. Detecting faces, trading stocks, maximizing some metric or other. So if super-intelligent AI is invented, it will probably seek to fulfill a ... (read more)
Imagine a man who really likes the color green. Maybe he's obsessed with it, to the exclusion of everything else, at a pathological extreme. This man doesn't seem too dangerous to us. However, what if the man were a genius? Then, due to his bizarre preferences, he becomes dangerous. Maybe he wants to turn the entire sky green, so he invents a smog generator that blots out the sun. Maybe he wants to turn people green, so he engineers a bioweapon that turns its victims green. High intelligence, plus a simple goal, is a recipe for disaster. Right now, compani... (read more)
Our human instinct to jump at a simple safeguard: “Aha! We’ll just unplug the [superhuman AI],” sounds to the [superhuman AI] like a spider saying, “Aha! We’ll kill the human by starving him, and we’ll starve him by not giving him a spider web to catch food with!” We’d just find 10,000 other ways to get food—like picking an apple off a tree—that a spider could never conceive of. (Quoted directly from here). (Policymakers)
Google, OpenAI, and other groups are working to create AI, smarter than any human at every mental task. But there's a problem: they're using their current "AI" software for narrow tasks. Recognizing faces, completing sentences, playing games. Researchers test things that are easy to measure, not what's necessarily best by complicated human wants and needs. So the first-ever superhuman AI, will probably be devoted to a "dumb" goal. If it wants to maximize its goal, it'll use its intelligence to steamroll the things humans value, and we likely couldn't stop ... (read more)
Machine learning researchers know how to keep making AI smarter, but have no idea where to begin with making AI loyal.
Most species in history have gone extinct. They get wiped out by predators, asteroids, human developments, and more. Once a species is extinct, it stays that way. What if a species could protect itself from these threats? Could it develop itself to a capacity where it can't go extinct? Can a species escape the danger permanently?
As it turns out, humanity may soon lurch towards one of those two fates. Extinction, or escape. The world's smartest scientists are working to create artificial intelligence, smarter than any human at any mental task. If such AI we... (read more)
I have a rigorous position paper on this very topic that addresses not only all audiences, but also is written with the interest layman &/or
Computers can already "think" faster than humans. If we created AI software that was smarter than humans, it would think better, not just faster. Giving a monkey more time won't necessarily help it learn quantum physics, because the monkey's mind may not have the capacity to understand the concept at all. Since there's no clear upper limit to how smart something can be, we'd expect superhumanly-smart AI to think on a level we can't comprehend. Such an AI would be unfathomably dangerous and hard to control. (Adapted from here). (Policymakers)
AI already can do thousands or even millions of tasks per second. If we invent a way for an AI to have thoughts as complicated or nuanced as a human, and plug it into an existing AI, it might be able to have a thousand or a million thoughs per second. That's a very dangerous thing.
Techxecutives and policymakers.
Some figures within machine learning have argued that the safety of broad-domain future AI is not a major concern. They argue that since narrow-domain present-day AI is already dangerous, this should be our primary concern, rather than that of future AI. But it doesn't have to be either/or.
Take climate change. Some climate scientists study the future possibilities of ice shelf collapses and disruptions of global weather cycles. Other climate scientists study the existing problems of more intense natural disasters and creeping desertification. But these two... (read more)
The future is not a race between AI and humanity. It's a race between AI safety and AI disaster.
(Policymakers, tech executives)
We need to be proactive about AI safety, not reactive.
(Policymakers)
If an artificial intelligence program became generally smarter than humans, there would be a massive power imbalance between the AI and humanity. Humans are slightly smarter than apes, yet we built a technological society while apes face extinction. Humans are much smarter than ants, and we barely think of the anthills we destroy to build highways. At a high enough level of intelligence, an AI program would be to us as we are to ants. (Adapted from here). (Policymakers)
One of the main concerns about general AI is that it could quickly get out of human control. If humans invent an AI with human-level cognitive skills, that AI could still think faster than humans, solve problems more precisely, and copy its own files to more computers. If inventing human-level AI is within human abilities, it's also within human-level-AI's abilities. So this AI could improve its own code, and get more intelligent over several iterations. Eventually, we would see a super-smart AI with superhuman mental abilities. Keeping control of that software could be an insurmountable challenge. (Adapted from here). (Policymakers)
Could AI technology really get as good as humans, at everything? There are multiple plausible ways for this to happen. As computing power increases every few years, we can simulate bigger "neural networks", which get smarter as they grow. A "narrow" AI could get good at AI programming, then program itself to get better at other tasks. Or perhaps some new algorithm or technique is waiting to be discovered, by some random researcher somewhere in the world, that "solves" intelligence itself. With all these paths to general AI, it could happen within our own l... (read more)
Right now, "AI" technologies are in use all around us. Google Translate uses AI to convert words from one language to another. Amazon uses AI to recommend products based on your purchase history. Self-driving cars use AI to detect objects in view. These are all "narrow" AI programs, used only for a specific task. Researchers at the top AI labs, however, are increasingly looking at "general" AI, that can do multiple different tasks. In other words, the field is trying to replicate the generalist abilities of humans, with software. (Partly adapted from here)... (read more)
Imagine a person from the year 1400 AD, being taken to the world of 1700. They would be shocked at the printing press, the telescope, the seafaring empires. Yet they would still find their footing. Now, imagine someone from 1700 being taken to 2022. To anyone from the era of horses and sailboats, our everyday cars and skyscrapers and smartphones would seem overwhelmingly magical. This implies something interesting: not only is the future usually weirder than the past, but it's getting weirder faster. This implies that, a century or two from now, the world may have changed beyond recognition. (Adapted from here). (Policymakers)
"Human minds represent a tiny dot in the vast space of all possible mind designs, and very different kinds of minds are unlikely to share to complex motivations unique to humans and other mammals."
https://intelligence.org/ie-faq/
ML researchers
We need to solve the problem of erratic behavior in AI, before erratic behavior emerges in an AI smart enough to identify and compromise human controllers.
(Policymakers) We have a good idea of what make bridges safe, through physics, materials science and rigorous testing. We can anticipate the conditions they'll operate in.
The very point of powerful AI systems is to operate in complex environments better than we can anticipate. Computer science can offer no guarantees if we don't even know what to check. Safety measures aren't catching up quickly enough.
We are somehow tolerating the mistakes of current AI systems. Nothing's ready for the next scale-up.
In the Soviet Union, there was a company that made machinery for vulcanizing rubber. They had the option to make more efficient machines, instead of their older models. However, they didn't do it, because they wouldn't get paid as much for making the new machines. Why would that be? Wouldn't more efficient machines be more desirable?
Well, yes, but the company got paid per pound of machine, and the new machines were lighter.
Now, you may say that this is just a problem with communist economies. Well, capitalist economies fall into very similar traps. If a co... (read more)
Policymakers and techxecutives, not ML researchers
With the emergence of Covid variants and social media, global quality of life/living conditions will improve and decline. It will ebb and flow, like the tide, and generally be impossible to prevent.
But if AI becomes as smart to humans as humans are to ants, that won't matter anymore. It will be effortless and cheap to automatically generate new ideas or new inventions, and give people whatever they want or need. But if the AI malfunctions instead, it would be like a tidal wave.
There is an enormous amount of joy, fulfillment, exploration, discovery, and prosperity in humanity's future... but only if advanced AI values those things.
(Policymakers, tech executives)
(ML researchers) We still don't have a robust solution to specification gaming: powerful agents find ways to get high reward, but not in the way you'd want. Sure, you can tweak your objective, add rules, but this doesn't solve the core problem, that your agent doesn't seek what you want, only a rough operational translation.
What would a high-fidelity translation would look like? How would create a system that doesn't try to game you?
Techxecutives and policymakers:
We have no idea when it will be invented. All we know is that it won't be tomorrow. But when we do discover that it's going to be tomorrow, it will already be too late to make it safe and predictable.
It is inevitable that humanity will one day build an AI that is smarter than a human. Engineers have been succeeding at that for decades now. But we instinctively think that such a day is centuries away, and that kind of thinking has always failed to predict every milestone that AI has crossed over the last 10 years.
"Human intelligence did not evolve in order to conquer the planet or explore the solar system. It emerged randomly, out of nowhere, as a byproduct of something much less significant."
Alternative:
Human intelligence did not evolve in order to conquer the planet or explore the solar system. It emerged randomly, out of nowhere, and without a single engineer trying to create it. And now we have armies of those engineers"
Refined for policymakers and techxecutives:
"A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, ... perform scientific experiments to understand the world better than humans can, ... manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to."
https://intelligence.org/ie-faq/
"A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, take over mobile machines connected to networks connected to the internet, use them to build additional machines, perform scientific experiments to understand the world better than humans can, invent quantum computing and nanotechnology, manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to."
https://intelligence.org/ie-faq/
"One might say that “Intelligence is no match for a gun, or for someone with lots of money,” but both guns and money were produced by intelligence. If not for our intelligence, humans would still be foraging the savannah for food."
https://intelligence.org/ie-faq/
"Machines are already smarter than humans are at many specific tasks: performing calculations, playing chess, searching large databanks, detecting underwater mines, and more. But one thing that makes humans special is their general intelligence. Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them."
https://intelligence.org/ie-faq/
"The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position"
Bostrom, superintelligence, 2014
For policymakers
Optional extra (for all 3):
"Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so t... (read more)
Look how much we suffered from a stupid, replicating code with goals adverse to ours; now, imagine how bad it would be if we have an intelligent replicating enemy...
(Tech execs) "Don’t ask if artificial intelligence is good or fair, ask how it shifts power". As a corollary, if your AI system is powerful enough to bypass human intervention, it surely won't be fair, nor good.
(ML researchers) Most policies are unsafe in a large enough search space; have you designed yours well, or are you optimizing through a minefield?
(Policymakers) AI systems are very much unlike humans. AI research isn't trying to replicate the human brain; the goal is, however, to be better than humans at certain tasks. For the AI industry, better means cheaper, faster, more precise, more reliable. A plane flies faster than birds, we don't care if it needs more fuel. Some properties are important (here, speed), some aren't (here, consumption).
When developing current AI systems, we're focusing on speed and precision, and we don't care about unintended outcomes. This isn't an issue for most systems: a ... (read more)
(Tech execs) Tax optimization is indeed optimization under the constraints of the tax code. People aren't just stumbling on loopholes, they're actually seeking them, not for the thrill of it, but because money is a strong incentive.
Consider now AI systems, built to maximize a given indicator, seeking whatever strategy is best, following your rules. They will get very creative with them, not for the thrill of it, but because it wins.
Good faith rules and heuristics are no match for adverse optimization.
(ML researchers) Powerful agents are able to search through a wide range of actions. The more efficient the search, the better the actions, the higher the rewards. So we are building agents that are searching in bigger and bigger spaces.
For a classic pathfinding algorithm, some paths are suboptimal, but all of them are safe, because they follow the map. For a self-driving car, some paths are suboptimal, but some are unsafe. There is no guarantee that the optimal path is safe, because we really don't know how to tell what is safe or not, yet.
A more efficient search isn't a safer search!
(Policymakers) The goals and rules we're putting into machines are law to them. What we're doing right now is making them really good at following the letter of this law, but not the spirit.
Whatever we really mean by those rules, is lost on the machine. Our ethics don't translate well. Therein lies the danger: competent, obedient, blind, just following the rules.
Even if you don't assume that the long-term future matters much, preventing AI risk is still a valuable policy objective. Here's why.
In regulatory cost-benefit analysis, a tool called the "value of a statistical life" is used to measure how much value people place on avoiding risks to their own life (source). Most government agencies, by asking about topics like how much people will pay for safety features in their car or how much people are paid for working in riskier jobs, assign a value of about ten million dollars to one statistical life. That is, redu... (read more)
AI might be nowhere near human-level yet. We're also nowhere near runaway climate change, but we still care about it.
(policymakers, tech executives)
"Follow the science" doesn't just apply to pandemics. It's time to listen to AI experts, not AI pundits.
(policymakers)
[Intended for Policymakers with the focus of simply allowing for them to be aware of the existence of AI as a threat to be taken seriously through an emotional appeal; Perhaps this could work for Tech executives, too.
I know this entry doesn't follow what a traditional paragraph is, but I like its content. Also it's a tad bit long, so I'll attach a separate comment under this one which is shorter, but I don't think it's as impactful]
Timmy is my personal AI Chef, and he is a pretty darn good one, too.
You pick a cuisine, and he mentally simulates himsel... (read more)
For policymakers: "Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all."
— Stephen Hawking, Stuart Russell, Max Tegmark, and Frank Wilczek (https://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence-but-are-we-taking-ai-seriously-enough-9313474.html)
For policymakers: "[AGI] could spell the end of the human race. […] it would […] redesign itself at an ever-increasing rate. Humans […] couldn't compete and would be superseded."
— Stephen Hawking (https://www.bbc.co.uk/news/technology-30290540)
For policymakers: "If people design computer viruses, someone will design AI that replicates itself. This will be a new form of life that will outperform humans."
— Stephen Hawking (https://www.wired.co.uk/article/stephen-hawking-interview-alien-life-climate-change-donald-trump)
YouTubers live in constant fear of the mysterious, capricious Algorithm. There is no mercy or sense, just rituals of appeasement as it maximizes "engagement." Imagine that, but it runs your whole life.
<Optional continuation:> You don't shop at Hot Topic because you hear it can hurt your ranking, which could damage your next hiring opportunity. And you iron your clothes despite the starch making you itch because it should boost your conscientiousness score, giving you an edge in dating apps.
Also this entire post by Duncan Sabien
(@ Tech Executives, Policymakers & Researches)
Back in February 2020, the vast majority of people didn't see the global event of Covid coming, even though all the signs were there. All it took was a fresh look at the evidence and some honest extrapolation.
Looking at recent AI progress, it seems very possible that we're in the "February of 2020" of AI.
(original argument by Duncan Sabien, rephrased)
(@ Tech Executives, Policymakers & Researches)
Tech Executives
You already know exponential growth. It’s a cliché at this point. Rice on a chessboard. Covid. Sequencing the human genome. Your business plan. But if you honestly stop and squint at the last few years of AI progress, and think about the amount of progress made in the many decades before, how sure are you really that AI is not on this trajectory? Would you bet your company on it? Your life? The world?
They say the pen is mightier than the sword. What about a million billion typewriters? AI can do that. We should make sure it's used for good. (policymaker)
AI will at one point be smarter than humans, smart enough to have power over us. How do we make sure we can trust it with the fate of everyone we love? (policymakers)
Tech Executives
“Move Fast and Break Things” doesn’t work when the thing that breaks is the world.
Policymakers
It only takes an AI about as complicated as an insect to keep you refreshing Twitter all day. What do you think future AIs could do to us?
Policymakers
A scorpion asks a frog to carry it across a river. The frog hesitates, afraid that the scorpion might sting it, but the scorpion promises not to, pointing out that it would drown if it killed the frog in the middle of the river. The frog agrees to transport the scorpion. Midway across the river, the scorpion stings the frog anyway, dooming them both. The dying frog asks the scorpion why it stung despite knowing the consequence, and the scorpion replies: "I am sorry, but I couldn't resist the urge. It's in my nature."
-Fable
We are the frog, and the nature of our future AI scorpions must be figured out or we all may die, or worse.
The more powerful a tool is, the more important it is that the tool behaves predictably.
A chainsaw that behaves unpredictably is very, very, dangerous.
AI is, conservatively, thousands of times more powerful than a chainsaw.
And unlike an unpredictable chainsaw, there is no guarantee we will be able to turn an unpredictable AI off and fix it or replace it.
It is plausible that the danger of failing to align AI safely - to make it predictable - is such that we only have one chance to get it right.
Finally, it is absurdly cheap to make massive progress in AI safety.
This is targeted at all 3 groups:
There is a time coming in the next decade or so when we will have released a veritable swarm of different genies that are able to understand and improve themselves better than we can. At that point, the genies will not being going back in the bottle, so we can only pray they like us.
Already we have turned all of our critical industries, all of our material resources, over to these . . . things . . . these lumps of silver and paste we call nanorobots. And now we propose to teach them intelligence? What, pray tell, will we do when these little homunculi awaken one day and announce that they have no further need of us?
— Sister Miriam Godwinson, "We Must Dissent"
Dogs and cats don't even know that they're going to die. But we do.
Imagine if an AI was as smart relative to humans, as humans are relative to dogs and cats. What would it know about us that we can't begin to understand?
On the day that AI becomes smarter than humans, it might do something horrible. We've done horrible things to less intelligent creatures, like ants and lions.
Longer version:
On the day that AI becomes smarter than humans, it might do something strange or horrible to us. We've done strange and horrible things to less intelligent creatures, like chickens, ants, dogs, tigers, and cows. They barely understand that we exist, let alone how or why we do awful things to them.
For Policymakers
People already mount today's AI on nuclear weapons. Imagine what they'll do when they find out that smarter-than-human AI is within reach.
Optional Extra: It could be invented in 10 years; nobody can predict when the the first person will figure it out.
Do we serve The Algorithm, or does it serve us? Choose before The Algorithm chooses for you.
That kid who always found loopholes in whatever his parents asked him? He made an AI that's just like him.
In order to remain stable, bureaucracies are filled with arcane rules, mazes, and traps that ensnare the uninitiated. This is necessary; there are hordes of intelligent opportunists at the gates, thinking of all sorts of ways to get in, take what they want, and never look back. But no matter how insulated the insiders are from the outsiders, someone always gets through. Life finds a way.
No matter how smart they appear, these are humans. If AI becomes as smart to humans as humans are to ants, it will be effortless.
For policymakers.
COVID and AI grow exponentially. In December 2019, COVID was a few people at a fish market. In January, it was just one city. In March, it was the world. In 2010, computers could beat humans at Chess. In 2016, at Go. In 2022, at art, writing, and truck driving. Are we ready for 2028?
Someone who likes machines more than people creates a machine to improve the world. Will the "improved" world have more people?
Humanity's incompetence has kept us from destroying ourselves. With AI, we will finally break that shackle.
The problem might even be impossible to solve, no matter how many PhD scholars we throw at it. So if we're going to have a rapid response team, ready to fix up the final invention as soon as it is invented, then we had better make sure that there's enough of them to get it done right.
For policymakers
Optional addition:
...final invention as soon as it's invented (an AI system that can do anything a human can, including inventing the next generation of AI systems that are even smarter), then we had better make sure that...
When asked about the idea of an AI smarter than a human, people tend to say "not for another hundred years". And they've been saying the exact same thing for 50 years now. The same thing happened with airplanes, the discovery of the nervous system, computers, and nuclear bombs; and often within 5 years of the discovery. And the last three years of groundbreaking progress in AI has made clear that they've never been more wrong.
"If you have an untrustworthy general superintelligence generating [sentences] meant to [prove something], then I would not only expect the superintelligence to be [smart enough] to fool humans in the sense of arguing for things that were [actually lies]... I'd expect the superintelligence to be able to covertly hack the human [mind] in ways that I wouldn't understand, even after having been told what happened[, because a superintelligence is, by definition, at least as smart to humans as humans are to chimpanzees]. So you must have some belief about the s... (read more)
Retracted
GPT-3 told me the following: Super intelligent AI presents a very real danger to humanity. If left unchecked, AI could eventually surpass human intelligence, leading to disastrous consequences. We must be very careful in how we develop and control AI in order to avoid this outcome.
GPT-3 told me the following: Super intelligent AI presents a very real danger to humanity. If left unchecked, AI could eventually surpass human intelligence, leading to disastrous consequences. We must be very careful in how we develop and control AI in order to avoid this outcome
There is a certain strain of thinker who insists on being more naturalist than Nature. They will say with great certainty that since Thor does not exist, Mr. Tesla must not exist either, and that the stories of Asclepius disprove Pasteur. This is quite backwards: it is reasonable to argue that a machine will never think because the Mechanical Turk couldn't; it is madness to say it will never think because Frankenstein's monster could. As well demand that we must deny Queen Victoria lest we accept Queen Mab, or doubt Jack London lest we admit Jack Frost. Na... (read more)
You might think that AI risk is no big deal. But would you bet your life on it?
(policymakers)
Betting against the people who said pandemics were a big deal, six years ago, is a losing proposition.
(policymakers, tech executives)
(source)
Just because tech billionaires care about AI risk doesn't mean you shouldn't. Even if a fool says the sky is blue, it's still blue.
(policymakers, maybe ML researchers)
Hoping you'll run out of gas before you drive off a cliff is a losing strategy. Align AGI; don't count on long timelines.
(ML researchers)
(adapted from Human Compatible)
Intelligence has an inverse relationship with predictability. Cranking up an AI's intelligence might crank down its predictability. Perhaps it will plummet at an unpredictable moment.
For policymakers, but also the other two. It's hard to handle policymakers, because many policymakers might think that an AGI would fly to the center of the galaxy and duke it out with God/Yaweh, mano a mano.
If the media reported on other dangers like it reported on AI risk, it would talk about issues very differently. It would compare events in the Middle East to Tom Clancy novels. It would dismiss runaway climate change by saying it hasn't happened yet. It would think of the risk of nuclear war in terms of putting people out of work. It would compare asteroid impacts to mudslides. It would call meteorologists "nerds" for talking about hurricanes.
AI risk is serious, and it isn't taken seriously. It's time to look past the sound bites and focus on what e... (read more)
Hey Siri, "Is there a God?" "There is now."
- Adapted from Fredric Brown, "The Answer" - for policymakers.
Humans are unpredictable. A hyperintelligent machine would notice this and see human control as fundamentally suboptimal, no matter the specifics of what's running under the hood.
AIs are like genies: they’ll fulfill the literal words of your wish, but don’t care at all about the spirit.
Once AI is close enough to human intelligence, it will be able to improve itself without human maintenance. It will be able to take itself the rest of the way, all the way up to humanlike intelligence, and it will probably pass that point as quickly as it arrived. There's no upper limit to intelligence, only an upper limit for intelligent humans; we don't know what a hyperintelligent machine would do, it's never happened before. If it had, we might not be alive right now; we simply don't know how something like that would behave, only that it would be as capable of outsmarting us as we are of outsmarting lions and hyenas.
Policymakers, and maybe techxecutives.
People said man wouldn't fly for a million years. Airplanes were fighting each other eleven years later. Superintelligent AI might happen faster than you think. (policymakers, tech executives) (source) (other source)
What if there was a second intelligent species on Earth, besides humans? The world's largest tech companies are working to build super-human AI, so we may find out sooner than you think. (Policymakers)
[open to critiques/rewordings so we don't accidentally ignore pretty-intelligent nonhuman animals, like dolphins.]
If we build an AI that's as smart as a human, that's not a human. There will be all sorts of things we did wrong, including deliberately, in order to optimize for performance. All someone has to do is crank it up 10x faster in order to get slightly better results, and that could be it. Something smarter than any human, thinking hundreds or millions of times faster than a human mind, able to learn or figure out anything it wants, including how to outmaneuver all human creators.
Techxecutives and ML researchers. I'd worry about sending this to policymakers, b... (read more)
We may be physically weaker than apes, but we're much smarter, so they don't really threaten us. Now, if a computer was much smarter than us... (Policymakers)
Superhuman, super-smart AI may seem like science fiction… but so did rockets and smartphones, once upon a time. (Policymakers, Tech executives)
If we can't keep advanced AI safe for us, it will keep itself safe from us... and we probably won't be happy with the result. (policymakers)
One of my one-liners was written by an AI. You probably can't tell which one. Imagine what AI could do once it's even more powerful. (tech executives, ML researchers)
AI could be the best or worst thing to ever happen to humanity. It all depends on how we design and program them. (policymakers, tech executives)
Building advanced AI without knowing how to align it is like launching a rocket without knowing how gravity works. (tech executives, ML researchers)
(based on MIRI's "The Rocket Alignment Problem")
When AI’s smarter than we are, we have to be sure it shares our values: after all, we won’t understand how it thinks. (policymakers, tech executives)
Social media AI has already messed up politics, and AI's only getting more powerful from here on out. (policymakers)
In the same way, suppose that you take weak domains where the AGI can't fool you, and apply some gradient descent to get the AGI to stop outputting actions of a type that humans can detect and label as 'manipulative'. And then you scale up that AGI to a superhuman domain. I predict that deep algorithms within the AGI will go through consequentialist dances, and model humans, and output human-manipulating actions that can't be detected as manipulative by the humans, in a way that seems likely to bypass whatever earlier patch was imbued by gradie... (read more)
"Manipulating humans is a [winning] strategy if you've accurately modeled (even at quite low resolution) what humans are and what they do in the larger scheme of things."
EY
For Techxecutives, maybe ML researchers, depends on what they work on.
"I'm doubtful that you can have an AGI that's significantly above human intelligence in all respects, without it having the capability-if-it-wanted-to of looking over its own code and seeing lots of potential improvements."
EY
Techxecutives, maybe ML researchers
For the last 20 years, AI technology has improved randomly and without warning. We are now closer to human level artificial intelligence than ever before, to building something that can invent solutions to our problems, or invent a way to make itself as smart to humans as humans are to ants. But we won't know what it will look like until after it is invented; if it is as smart as a human but learns one thousand times as fast, it might detect the control system and compromise all human controllers. Choice and intent are irrelevant; it's a computer, all it t... (read more)
If an AI system becomes as smart to a human as a human is to a lion or cow, it will not have much trouble compromising its human controllers. After a certain level of intelligence, above human-level intelligence, AGI could thwart any manmade control system immediately after a single glitch, the same way that humans do things on a whim.
For policymakers, maybe techxecutives too.
An AI capable of programming might be able to reprogram itself smarter and smarter, soon surpassing humans as the dominant species on the planet.
I just won an election using nothing but AI-written speeches
"When we create an Artificial General Intelligence, we will be giving it the power to fundamentally transform human society, and the choices that we make now will affect how good or bad those transformations will be. In the same way that humanity was transformed when chemist and physicists discovered how to make nuclear weapons, the ideas developed now around AI alignment will be directly relevant to shaping our future."
"Once we make an Artificial General Intelligence, it's going to try and achieve its goals however it can, including convincing everyone around it that it should achieve them. If we don't make sure that it's goals are aligned with humanity's, we won't be able to stop it."
"The present situation can be seen as one in which a common resource, the remaining timeline until AGI shows up, is incentivized to be burned by AI researchers because they have to come up with neat publications and publish them (which burns the remaining timeline) in order to earn status and higher salaries"
Eliezer Yudkowsky
Bear in mind that in reality, there is a demand for cutting edge AI to be mounted on nuclear cruise missiles, and that no person, group, or movement can prevent the military from procuring the nuclear weapons technology that it claims ... (read more)
50 years ago was 1970. The gap between AI systems then and AI systems now seems pretty plausibly greater than the remaining gap [between current AI and AGI],
Nate Soares
Optional extra:
even before accounting the recent dramatic increase in the rate of progress, and potential future increases in rate-of-progress as it starts to feel within-grasp.
"There are 7 billion people on this planet. Each one of them has different life experiences, different desires, different aspirations, and different values. The kinds of things that would cause two of us to act, could cause a third person to be compelled to do the opposite. An Artificial General Intelligence will have no choice but to act from the goals we give it. When we give it goals that 1/3 of the planet disagrees with, what will happen next?" (Policy maker)
All our ML systems optimize for some proxy of what we really want, like "high score" for playing a game well, or "ads clicked" for suggesting news sites. As AI gets better at optimizing the high score, the things imperfectly linked with the high score (like good gameplay or accurate journalism) get thrown aside. After all, it's not efficient to maximize a value that wasn't programmed in. (Machine learning researchers)
The big ML systems, like GPT-3 and MT-NLG, are taking increasingly large amounts of computing power to train. Normally, this requires expensive GPUs to do math simulating neural networks. But the human brain, better than any computer at a variety of tasks, is fairly small. Plus, it uses far less energy than huge GPU clusters!
As much as AI has progressed in the last few years, is it headed for an AI winter? Not with new hardware on the horizon. By mimicking the human brain, neuromorphic chips and new analog computers can do ML calculations faster than GPUs,... (read more)
"With all the advanced tools we have, and with all our access to scientific information, we still don't know the consequences of many of our actions. We still can't predict weather more than a few days out, we can't predict what will happen when we say something to another person, we can't predict how to get our kids to do what we say. When we create a whole new type of intelligence, why should we be able to predict what happens then?"
"There are lots of ways to be smarter than people. Having a better memory, being able to learn faster, having more thorough reasoning, deeper insight, more creativity, etc.
When we make something smarter than ourselves, it won't have trouble doing what we ask. But we will have to make sure we ask it to do the right thing, and I've yet to hear a goal everyone can agree on." (Policy maker)
Many in Silicon Valley (and China) think there's a huge advantage to inventing super-smart AI first, so they can reap the benefits and "pull ahead" of competitors. If an AI smarter than any human can think of military strategies, or hacking techniques, or terrorist plots, this is a terrifying scenario. (Policymakers)
What will happen if an AI realises that it is in a training loop? There are a lot of bad scenarios that could branch out from this point. Potentially this scenario sounds weird or crazy, however, humans possess the ability to introspect and philosophise on similar topics, even though our brains are “simply” computational apparatus which do not seem to possess any qualities that a sufficiently advanced AI could not possess.
"When we start trying to think about how to best make the world a better place, rarely can anyone agree on what is the right way. Imagine what would happen to if the first person to make an Artificial General intelligence got to say what the right way is, and then get instructed how to implement it like they had every researcher and commentator alive spending all their day figuring out the best way to implement it. Without thinking about if it was the right thing to do." (Policymaker)
"How well would your life, or the life of your child go, if you took your only purpose in life to be the first thing your child asked you to do for them? When we make an Artificial Intelligence smarter than us, we will be giving it as well defined of a goal as we know how. Unfortunately, the same as a 5 year old doesn't know what would happen if their parent dropped everything to do what they request, we won't be able to think through the consequences of what we request." (Policymaker)
"All scientific ignorance is hallowed by ancientness. Each and every absence of knowledge dates back to the dawn of human curiosity; and the hole lasts through the ages, seemingly eternal, right up until someone fills it."
EY, AI as a pos neg factor, 2006ish
Optional extra:
"I think it is possible for mere fallible humans to succeed on the challenge of building Friendly AI. But only if intelligence ceases to be a sacred mystery to us, as life was a sacred mystery to [scientists in 1922]. Intelligence must cease to be any kind of mystery whatever, sacred or not. We must execute the creation of Artificial Intelligence as the exact application of an exact art"
"Intelligence is not the first thing human science has ever encountered which proved difficult to understand. Stars were once mysteries, and chemistry, and biology. Generations of investigators tried and failed to understand those mysteries, and they acquired the reputation of being impossible to mere science"
EY, AI as a pos neg factor, 2006ish
Optional extra:
"Once upon a time, no one understood why some matter was inert and lifeless, while other matter pulsed with blood and vitality. No one knew how living matter reproduced itself, or why our hands obeyed our mental orders."
"The human mind is not the limit of the possible. Homo sapiens represents the first general intelligence. We were born into the uttermost beginning of things, the dawn of mind."
EY, AI as a pos neg factor, 2006ish
All three categories (most of my submissions go here, unless otherwise stated)
"It seems like an unfair challenge. Such competence is not historically typical of human institutions, no matter how hard they try. For decades the U.S. and the U.S.S.R. avoided nuclear war, but not perfectly; there were close calls, such as the Cuban Missile Crisis in 1962."
EY, AI as a pos neg factor, 2006ish
For policymakers (#1) and techxecutives (#2)
"Hominids have survived this long only because, for the last million years, there were no arsenals of hydrogen bombs, no spaceships to steer asteroids toward Earth, no biological weapons labs to produce superviruses, no recurring annual prospect of nuclear war... [and soon, we may have to worry about AGI as well, the finish line]. To survive any appreciable time, we need to drive down each risk to nearly zero.
EY, AI as a pos neg factor, 2006ish
For ML researchers only, since policymakers and techxecutives are often accustomed to "survival of the fittest" lifestyles
"Often Nature poses requirements that are grossly unfair, even on tests where the penalty for failure is death. How is a 10th-century medieval peasant supposed to invent a cure for tuberculosis? Nature does not match her challenges to your skill, or your resources, or how much free time you have to think about the problem."
EY, AI as a pos neg factor, 2006ish
Optional extra:
And when you run into a lethal challenge too difficult for you, you die. It may be unpleasant to think about, but that has been the reality for humans, for thousands upon thousands ... (read more)
"If the pen is exactly vertical, it may remain upright; but if the pen tilts even a little from the vertical, gravity pulls it further in that direction, and the process accelerates. So too would smarter systems have an easier time making themselves smarter."
EY, AI as a pos neg factor
For policymakers, Techxecutives, and only ML researchers who are already partially sold on the "singularity" concept (or have the slightest interest in it).
Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.
I.J. Good, 1965 (he is dead and I don't have any claim to his share, even if he gets the entire share). FYI he was one of the first computer scientists and later on, he w... (read more)
If AI safety were a scam to raise money… it'd have way better marketing! (Adapted from here). (Machine learning researchers)
Every headline about an AI improvement is another warning of superhuman intelligence, and as a species, we're not ready for it. Crunch time is near. (Adapted from here). (Machine learning researchers)
Getting AI to understand human values, when we ourselves barely understand them, is a massive challenge for the world's best computer scientists, mathematicians, and philosophers. Are you one of them? (Adapted from here). (Machine learning researchers)
If AI can eventually be made smarter than humans, our fate would depend on it in the same way Earth's species depend on us. This is not reassuring for us. (Adapted from here). (Policymakers)
Humans are the smartest species on Earth… but we might still be the dumbest species that could've possibly built our technology. (Adapted from here). (Tech executives)
What are we supposed to do if we build something that's so vastly smarter than us that it can avoid being turned off? Certainly not turn it off.
What are we supposed to do with a worthless block that wants to be turned off, or randomly thinks up some reason to turn itself off? We try again.
"Who can say what science does not know? There is far too much science for any one human being to learn. Who can say that we are not ready for a scientific revolution, in advance of the surprise? And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI. Those two statements are not at all equivalent!"
EY, AI as a pos neg factor, 2006ish
"The «ten-year rule» for genius, validated across fields ranging from math to music to competitive tennis, states that no one achieves outstanding performance in any field without at least ten years of effort. (Hayes 1981.) ... If we want people who can make progress on Friendly AI, then they have to start training themselves, full-time, years before they are urgently needed."
EY, AI as a pos neg factor, 2006ish
When will superhuman AI be invented? Experts disagree, but their estimates mostly range from a few years to a few decades away. The consensus is that artificial general intelligence, capable of beating humans at any mental task, will probably arrive before the end of the century. But, again, it could be just a few years away. With so much uncertainty, should't we start worrying about the new nonhuman intelligent beings now? (Adapted from here). (Machine learning researchers)
Right now, AI only optimizes for proxies of human values… is that good enough for the future? (Adapted from here). (Machine learning researchers)
The failed promises [still] come swiftly to mind, both inside and outside the field of AI, when [AGI] is mentioned. The culture of AI research has adapted to this condition: There is a taboo against talking about human-level capabilities. There is a stronger taboo against anyone who appears to be claiming or predicting a capability they have not demonstrated with running code.
EY, AI as a pos neg factor, 2006ish
Policymakers and techxecutives might misunderstand or be put on their guard by these, so ML researchers only.
"If you deal out cards from a deck, one after another, you will eventually deal out the ace of clubs."
"We keep dealing through the deck until we deal out an [AGI], whether it is the ace of hearts or the ace of clubs."
EY, AI as a pos neg factor, 2006ish
Sharp, unanticipated jumps in intelligence are proven possible, known to occur frequently, and have a long history of taking place"
Derived from:
"I cannot perform a precise calculation using a precisely confirmed theory, but my current opinion is that sharp jumps in intelligence arepossible, likely, and constitute the dominant probability."
EY, AI as a pos neg factor, 2006ish
For policymakers and techxecutives:
"When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent [systems] -- on a still-shorter time scale."
Optional extra (for same non-ML audience)
"The best analogy I see is to the evolutionary past: Animals can adapt to problems and make inventions, but often no faster than natural selection can do its work -- the world acts as its own simulator in the case of natural selection. We h... (read more)
"history books tend to focus selectively on movements that have an impact, as opposed to the vast majority that never amount to anything. There is an element involved of luck, and of the public’s prior willingness to hear."
EY, AI as a pos neg factor, 2006ish
For ML researchers, as policymakers might have their own ideas about how history books end up written.
"To date [in 1993], there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes," then there is little doubt that [even] more intelligent [machines] can be constructed shortly thereafter."
Vinge, technological singularity, 1993
"I argue that confinement is intrinsically impractical. Imagine yourself locked in your home with only limited data access to the outside, to your masters. If those masters thought at a rate -- say -- one million times slower than you, there is little doubt that over a period of years (your time) you could come up with a way to escape. I call this "fast thinking" form of superintelligence "weak superhumanity." Such a "weakly superhuman" entity would probably burn out in a few weeks of outside time. "Strong superhumanity" would be more than cranking up the ... (read more)
"a wolf cannot understand how a gun works, or what sort of effort goes into making a gun, or the nature of that human power which lets us invent guns"
Optional extra (for techxecutives and policymakers)
"Vinge (1993) wrote:
Strong superhumanity would be more than cranking up the clock speed on a human-equivalent mind. It’s hard to say precisely what strong superhumanity would be like, but the difference appears to be profound. Imagine running a dog mind at very high speed. Would a thousand years of doggy living add up to any human insight?"
EY, AI as a pos neg factor, 2006ish
Here is an example of a way that something could kill everyone if it suddenly became significantly smarter than a human:
- Crack the protein folding problem, to the extent of being able to generate DNA strings whose folded peptide sequences fill specific functional roles in a complex chemical interaction.
- Email sets of DNA strings to one or more online laboratories which offer DNA synthesis, peptide sequencing, and FedEx delivery. (Many labs currently offer this service, and some boast of 72-hour turnaround times.)
- Find at least one human connected to the Inter
... (read more)"The future has a reputation for accomplishing feats which the past thought impossible"
Optional extra:
"Future civilizations have even broken what past civilizations thought (incorrectly, of course) to be the laws of physics. If prophets of 1900 AD — never mind 1000 AD — had tried to bound the powers of human civilization a billion years later, some of those impossibilities would have been accomplished before the century was out; transmuting lead into gold, for example"
EY, AI as a pos neg factor, 2006ish
"We are separated from the risky [situation], not by large visible installations like isotope centrifuges or particle accelerators, but only by missing knowledge"
Optional extra:
"To use a perhaps over-dramatic metaphor, imagine if subcritical masses of enriched uranium had powered cars and ships throughout the world, before Leo Szilard first thought of the chain reaction"
EY, AI as a pos neg factor, 2006ish
"...computers are everywhere. It is not like the problem of nuclear proliferation, where the main emphasis is on controlling plutonium and enriched uranium. The raw materials for A[G]I are already everywhere. That cat is so far out of the bag that it’s in your wristwatch, cellphone, and dishwasher"
EY, AI as a pos neg factor, around 2006
The invention of the nuclear bomb took the human race by surprise. It seemed impossible. We did not know if it could be done until the moment one person discovered a way to do it, and in that moment it was invented. It was too late.
Optional extra, for ML researchers and techxecutives (policymakers may have all sorts of opinions about nukes):
Our survival has been totally dependent on good luck ever since.
If you work at Google DeepMind, or OpenAI, or Facebook AI, or Baidu, you may be at the cutting edge of AI research. Plus, your employer has the resources to scale up the systems you build, and to make sure they impact the lives of millions (or billions) of people. Any error in the AI's behavior, any bias in its training data, any human desideratum lost in optimization, will get magnified by millions or billions. You, personally, are in the best position to prevent this. (Machine learning researchers)
"In 1933, Lord Ernest Rutherford said that no one could ever expect to derive power from splitting the atom: «Anyone who looked for a source of power in the transformation of atoms was talking moonshine.» At that time laborious hours and weeks were required to fission a handful of nuclei"
Optional extra (for ML researchers):
"Flash forward to 1942, in a squash court beneath Stagg Field at the University of Chicago. Physicists are building a shape like a giant doorknob out of alternate layers of graphite and uranium, intended to start the first self-sustainin... (read more)
By the time smarter-than-human AI is invented, it could be too late to make sure it's safe for humanity. (Adapted from here). (Machine learning researchers)
"An Artificial [General] Intelligence could rewrite its code from scratch — it could change the underlying dynamics of optimization."
Optional extra (for ML researchers and techxecutives):
"Such an optimization process would wrap around much more strongly than either evolution accumulating adaptations, or humans accumulating knowledge. The key implication for our purposes is that an AI might make a huge jump in intelligence after reaching some threshold of criticality."
EY, AI as a pos neg factor, 2006ish
If you get a signal from space, that aliens will arrive in 30 years… do you wait 30 years, or do you start serious work now? (Adapted from here). (Policymakers)
"The human species, Homo sapiens, is a first mover. From an evolutionary perspective, our cousins, the chimpanzees, are only a hairbreadth away from us. Homo sapiens still wound up with all the technological marbles because we got there a little earlier."
Optional extra:
"Evolutionary biologists are still trying to unravel which order the key thresholds came in, because the first-mover species was first to cross so many: Speech, technology, abstract thought… We’re still trying to reconstruct which dominos knocked over which othe... (read m