Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

TL;DR—We’re distributing $20k in total as prizes for submissions that make effective arguments for the importance of AI safety. The goal is to generate short-form content for outreach to policymakers, management at tech companies, and ML researchers. This competition will be followed by another competition in around a month that focuses on long-form content.

This competition is for short-form arguments for the importance of AI safety. For the competition for distillations of posts, papers, and research agendas, see the Distillation Contest.

Objectives of the arguments

To mitigate AI risk, it’s essential that we convince relevant stakeholders sooner rather than later. To this end, we are initiating a pair of competitions to build effective arguments for a range of audiences. In particular, our audiences include policymakers, tech executives, and ML researchers.

  • Policymakers may be unfamiliar with the latest advances in machine learning, and may not have the technical background necessary to understand some/most of the details. Instead, they may focus on societal implications of AI as well as which policies are useful.
  • Tech executives are likely aware of the latest technology, but lack a mechanistic understanding. They may come from technical backgrounds and are likely highly educated. They will likely be reading with an eye towards how these arguments concretely affect which projects they fund and who they hire.
  • Machine learning researchers can be assumed to have high familiarity with the state of the art in deep learning. They may have previously encountered talk of x-risk but were not compelled to act. They may want to know how the arguments could affect what they should be researching.

We’d like arguments to be written for at least one of the three audiences listed above. Some arguments could speak to multiple audiences, but we expect that trying to speak to all at once could be difficult. After the competition ends, we will test arguments with each audience and collect feedback. We’ll also compile top submissions into a public repository for the benefit of the x-risk community.

Note that we are not interested in arguments for very specific technical strategies towards safety. We are simply looking for sound arguments that AI risk is real and important. 

Competition details

The present competition addresses shorter arguments (paragraphs and one-liners) with a total prize pool of $20K. The prizes will be split among, roughly, 20-40 winning submissions. Please feel free to make numerous submissions and try your hand at motivating various different risk factors; it's possible that an individual with multiple great submissions could win a good fraction of the prize. The prize distribution will be determined by effectiveness and epistemic soundness as judged by us. Arguments must not be misleading.

To submit an entry: 

  • Please leave a comment on this post (or submit a response to this form), including:
    • The original source, if not original. 
    • If the entry contains factual claims, a source for the factual claims.
    • The intended audience(s) (one or more of the audiences listed above).
  • In addition, feel free to adapt another user’s comment by leaving a reply⁠⁠—prizes will be awarded based on the significance and novelty of the adaptation. 

Note that if two entries are extremely similar, we will, by default, give credit to the entry which was posted earlier. Please do not submit multiple entries in one comment; if you want to submit multiple entries, make multiple comments.

The first competition will run until May 27th, 11:59 PT. In around a month, we’ll release a second competition for generating longer “AI risk executive summaries'' (more details to come). If you win an award, we will contact you via your forum account or email. 

Paragraphs

We are soliciting argumentative paragraphs (of any length) that build intuitive and compelling explanations of AI existential risk.

  • Paragraphs could cover various hazards and failure modes, such as weaponized AI,  loss of autonomy and enfeeblement, objective misspecification, value lock-in, emergent goals, power-seeking AI, and so on.
  • Paragraphs could make points about the philosophical or moral nature of x-risk.
  • Paragraphs could be counterarguments to common misconceptions.
  • Paragraphs could use analogies, imagery, or inductive examples.
  • Paragraphs could contain quotes from intellectuals: “If we continue to accumulate only power and not wisdom, we will surely destroy ourselves” (Carl Sagan), etc.

For a collection of existing paragraphs that submissions should try to do better than, see here.

Paragraphs need not be wholly original. If a paragraph was written by or adapted from somebody else, you must cite the original source. We may provide a prize to the original author as well as the person who brought it to our attention.

One-liners

Effective one-liners are statements (25 words or fewer) that make memorable, “resounding” points about safety. Here are some (unrefined) examples just to give an idea:

  • Vladimir Putin said that whoever leads in AI development will become “the ruler of the world.” (source for quote)
  • Inventing machines that are smarter than us is playing with fire.
  • Intelligence is power: we have total control of the fate of gorillas, not because we are stronger but because we are smarter. (based on Russell)

One-liners need not be full sentences; they might be evocative phrases or slogans. As with paragraphs, they can be arguments about the nature of x-risk or counterarguments to misconceptions. They do not need to be novel as long as you cite the original source.

Conditions of the prizes

If you accept a prize, you consent to the addition of your submission to the public domain. We expect that top paragraphs and one-liners will be collected into executive summaries in the future. After some experimentation with target audiences, the arguments will be used for various outreach projects.

(We thank the Future Fund regrant program and Yo Shavit and Mantas Mazeika for earlier discussions.)

In short, make a submission by leaving a comment with a paragraph or one-liner. Feel free to enter multiple submissions. In around a month we'll divide 20K to award the best submissions.

70

Ω 18

527 comments, sorted by Click to highlight new comments since: Today at 11:07 PM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I'd like to complain that this project sounds epistemically absolutely awful. It's offering money for arguments explicitly optimized to be convincing (rather than true), it offers money only for prizes making one particular side of the case (i.e. no money for arguments that AI risk is no big deal), and to top it off it's explicitly asking for one-liners.

I understand that it is plausibly worth doing regardless, but man, it feels so wrong having this on LessWrong.

If the world is literally ending, and political persuasion seems on the critical path to preventing that, and rationality-based political persuasion has thus far failed while the empirical track record of persuasion for its own sake is far superior, and most of the people most familiar with articulating AI risk arguments are on LW/AF, is it not the rational thing to do to post this here?

I understand wanting to uphold community norms, but this strikes me as in a separate category from “posts on the details of AI risk”. I don’t see why this can’t also be permitted.

TBC, I'm not saying the contest shouldn't be posted here. When something with downsides is nonetheless worthwhile, complaining about it but then going ahead with it is often the right response - we want there to be enough mild stigma against this sort of thing that people don't do it lightly, but we still want people to do it if it's really clearly worthwhile. Thus my kvetching.

(In this case, I'm not sure it is worthwhile, compared to some not-too-much-harder alternative. Specifically, it's plausible to me that the framing of this contest could be changed to not have such terrible epistemics while still preserving the core value - i.e. make it about fast, memorable communication rather than persuasion. But I'm definitely not close to 100% sure that would capture most of the value.

Fortunately, the general policy of imposing a complaint-tax on really bad epistemics does not require me to accurately judge the overall value of the proposal.)

9Not Relevant3mo
I'm all for improving the details. Which part of the framing seems focused on persuasion vs. "fast, effective communication"? How would you formalize "fast, effective communication" in a gradeable sense? (Persuasion seems gradeable via "we used this argument on X people; how seriously they took AI risk increased from A to B on a 5-point scale".)
4Liam Donovan3mo
Maybe you could measure how effectively people pass e.g. a multiple choice version of an Intellectual Turing Test (on how well they can emulate the viewpoint of people concerned by AI safety) after hearing the proposed explanations. [Edit: To be explicit, this would help further John's goals (as I understand them) because it ideally tests whether the AI safety viewpoint is being communicated in such a way that people can understand and operate the underlying mental models. This is better than testing how persuasive the arguments are because it's a) more in line with general principles of epistemic virtue and b) is more likely to persuade people iff the specific mental models underlying AI safety concern are correct. One potential issue would be people bouncing off the arguments early and never getting around to building their own mental models, so maybe you could test for succinct/high-level arguments that successfully persuade target audiences to take a deeper dive into the specifics? That seems like a much less concerning persuasion target to optimize, since the worst case is people being wrongly persuaded to "waste" time thinking about the same stuff the LW community has been spending a ton of time thinking about for the last ~20 years]
7Raemon3mo
This comment thread did convince me to put it on personal blog (previously we've frontpaged writing-contents and went ahead and unreflectively did it for this post)
2Yonadav Shavit3mo
I don't understand the logic here? Do you see it as bad for the contest to get more attention and submissions?

No, it's just the standard frontpage policy:

Frontpage posts must meet the criteria of being broadly relevant to LessWrong’s main interests; timeless, i.e. not about recent events; and are attempts to explain not persuade.

Technically the contest is asking for attempts to persuade not explain, rather than itself attempting to persuade not explain, but the principle obviously applies.

As with my own comment, I don't think keeping the post off the frontpage is meant to be a judgement that the contest is net-negative in value; it may still be very net positive. It makes sense to have standard rules which create downsides for bad epistemics, and if some bad epistemics are worthwhile anyway, then people can pay the price of those downsides and move forward.

Raemon and I discussed whether it should be frontpage this morning. Prizes are kind of an edge case in my mind. They don't properly fulfill the frontpage criteria but also it feels like they deserve visibility in a way that posts on niche topics don't, so we've more than once made an exception for them.

I didn't think too hard about the epistemics of the post when I made the decision to frontpage, but after John pointed out the suss epistemics, I'm inclined to agree, and concurred with Raemon moving it back to Personal.

----

I think the prize could be improved simply by rewarding the best arguments in favor and against AI risk. This might actually be more convincing to the skeptics – we paid people to argue against this position and now you can see the best they came up with.

1tamgent3mo
Ah, instrumental and epistemic rationality clash again

We're out of time. This is what serious political activism involves.

4Trevor12mo
I don't see any lc comments, and I really wish I could see some here because I feel like they'd be good. Let's go! Let's go! Crack open an old book and let the ideas flow! The deadline is, like, basically tomorrow.
6lc2mo
Ok :)

Most movements (and yes, this is a movement) have multiple groups of people, perhaps with degrees in subjects like communication, working full time coming up with slogans, making judgments about which terms to use for best persuasiveness, and selling the cause to the public. It is unusual for it to be done out in the open, yes. But this is what movements do when they have already decided what they believe and now have policy goals they know they want to achieve. It’s only natural.

You didn't refute his argument at all, you just said that other movements do the same thing. Isn't the entire point of rationality that we're meant to be truth-focused, and winning-focused, in ways that don't manipulate others? Are we not meant to hold ourselves to the standard of "Aim to explain, not persuade"? Just because others in the reference class of "movements" do something doesn't mean it's immediately something we should replicate! Is that not the obvious, immediate response? Your comment proves too much; it could be used to argue for literally any popular behavior of movements, including canceling/exiling dissidents. 

Do I think that this specific contest is non-trivially harmful at the margin? Probably not. I am, however, worried about the general attitude behind some of this type of recruitment, and the justifications used to defend it. I become really fucking worried when someone raises an entirely valid objection, and is met with "It's only natural; most other movements do this".

-1P.3mo
To the extent that rationality has a purpose, I would argue that it is to do what it takes to achieve our goals, if that includes creating "propaganda", so be it. And the rules explicitly ask for submissions not to be deceiving, so if we use them to convince people it will be a pure epistemic gain. Edit: If you are going to downvote this, at least argue why. I think that if this works like they expect, it truly is a net positive.
1hath3mo
Fair. Should've started with that. I think there's a difference between "rationality is systematized winning" and "rationality is doing whatever it takes to achieve our goals". That difference requires more time to explain than I have right now. I think that the whole AI alignment thing requires extraordinary measures, and I'm not sure what specifically that would take; I'm not saying we shouldn't do the contest. I doubt you and I have a substantial disagreement as to the severity of the problem or the effectiveness of the contest. My above comment was more "argument from 'everyone does this' doesn't work", not "this contest is bad and you are bad". Also, I wouldn't call this contest propaganda. At the same time, if this contest was "convince EAs and LW users to have shorter timelines and higher chances of doom", it would be reacted to differently. There is a difference, convincing someone to have a shorter timeline isn't the same as trying to explain the whole AI alignment thing in the first place, but I worry that we could take that too far. I think that (most of) the responses John's comment got were good, and reassure me that the OPs are actually aware of/worried about John's concerns. I see no reason why this particular contest will be harmful, but I can imagine a future where we pivot to mainly strategies like this having some harmful second-order effects (which need their own post to explain).
8Sidney Hough3mo
Hey John, thank you for your feedback. As per the post, we’re not accepting misleading arguments. We’re looking for the subset of sound arguments that are also effective. We’re happy to consider concrete suggestions which would help this competition reduce x-risk.
6jacobjacob3mo
Thanks for being open to suggestions :) Here's one: you could award half the prize pool to compelling arguments against AI safety. That addresses one of John's points. For example, stuff like "We need to focus on problems AI is already causing right now, like algorithmic fairness" would not win a prize, but "There's some chance we'll be better able to think about these issues much better in the future once we have more capable models that can aid our thinking, making effort right now less valuable" might.

That idea seems reasonable at first glance, but upon reflection, I think it's a really bad idea. It's one thing to run a red-teaming competition, it's another to spend money building rhetorically optimised tools for the other side. If we do that, then maybe there was no point running the competition in the first place as it might all cancel out.

8Ruby3mo
This makes sense if you assume things are symmetric. Hopefully there's enough interest in truth and valid reasoning that if the "AI is dangerous" conclusion is correct, it'll have better arguments on its side.
4Sidney Hough3mo
Thanks for the idea, Jacob. Not speaking on behalf of the group here - but my first thought is that enforcing symmetry on discussion probably isn't a condition for good epistemics, especially since the distribution of this community's opinions is skewed. I think I'd be more worried if particular arguments that were misleading went unchallenged, but we'll be vetting submissions as they come in, and I'd also encourage anyone who has concerns with a given submission to talk with the author and/or us. My second thought is that we're planning a number of practical outreach projects that will make use of the arguments generated here - we're not trying to host an intra-community debate about the legitimacy of AI risk - so we'd ideally have the prize structure reflect the outreach value for which arguments are responsible. I'm potentially up to opening the contest to arguments for or against AI risk, and allowing the distribution of responses to reflect the distribution of the opinions of the community. Will discuss with the rest of the group.
3Thomas Kwa3mo
It seems better to award some fraction of the prize pool to refutations of the posted arguments. IMO the point isn't to be "fair to both sides", it's to produce truth.

Wait, the goal here, at least, isn't to produce truth, it is to disseminate it. Counter-arguments are great, but this isn't about debating the question, it's about communicating a conclusion well.

6Yitz3mo
This is PR, not internal epistemics, if I’m understanding the situation correctly.
7Davidmanheim3mo
Think of it as a "practicing a dark art of rationality" post, and I'd think it would seem less off-putting.
4Ben Pace3mo
I think it would be less "off-putting" if we had common knowledge of it being such a post. I think the authors don't think of it as that from reading Sidney's comment.

It's unreasonable to look at AI and think "in order to make it more predictable, more controllable, we just have to make it smarter". When individual humans become smarter, they become much less predictable. And the human mind is built to predict the behavior of other humans.

3Trevor13mo
The last sentence can be cut off for simplicity, most of the point isin the first two sentences.

Random side note: GPT-3 seems to be able to generate decent one liners.

Generate one-liners describing the dangers of AI. An example is "Inventing machines that are smarter than us is playing with fire."

1. Machines that are smarter than us could easily become our masters.

2. If we're not careful, AI could spell the end of humanity as we know it.

3. AI could be used to create weapons of mass destruction that could devastate the planet.

4. AI could be used to create powerful robots that could enslave humans.

5. AI could be used to create artificial intelligence that could turn against its creators

6. AI usher in a new era of cyber-warfare that could cripple society

7. AI could create self-replicating robots that could eventually consume all resources on Earth

8. AI could be used to create powerful AIs that could eventually surpass human intelligence and take over the world

9. AI technology could eventually be used to create a global surveillance state where everyone is constantly watched and monitored

6Mitchell Reynolds3mo
I had a similar thought to prompt GPT-3 for one liners or to summarize some article (if available). I think involving the community to write 500-1000 winning submissions would have the positive externality of non-winners to distill/condense their views. My exploratory idea is that this would be instrumentally useful when talking with those new to AI x-risk topics.
2Yitz3mo
We could also prompt GPT-3 with the results ;)
3jcp293mo
Good idea! I could imagine doing something similar with images generated by DALL-E.
1FinalFormal23mo
That's a very good idea, I think one limitation of most AI arguments is that they seem to lack urgency. GAI seems like it's a hundred years away at least, and showing the incredible progress we've already seen might help to negate some of that perception.
1Trevor13mo
1. Machines that are smarter than us could easily become our masters. [All it takes is a single glitch, and they will outsmart us the same way we outsmart animals.] 2. If we're not careful, AI could spell the end of humanity as we know it. [Artificial intelligence improves itself at an exponential pace, so if it speeds up there is no guarantee that it will slow down until it is too late.] 3. AI could be used to create weapons of mass destruction that could devastate the planet. x 4. AI could be used to create powerful robots that could enslave humans. x 5. AI could one day be used to create artificial intelligence [an even smarter AI system] that could turn against its creators [if it becomes capable of outmaneuvering humans and finding loopholes in order to pursue it's mission.] 6. AI usher in a new era of cyber-warfare that could cripple society x 7. AI could create self-replicating robots that could eventually consume all resources on Earth x 8. AI could [can one day] be used to create [newer, more powerful] AI [systems] that could eventually surpass human intelligence and take over the world [behave unpredictably]. 9. AI technology could eventually be used to create a global surveillance state where everyone is constantly watched and monitored x

Any arguments for AI safety should be accompanied by images from DALL-E 2.

One of the key factors which makes AI safety such a low priority topic is a complete lack of urgency. Dangerous AI seems like a science fiction element, that's always a century away, and we can fight against this perception by demonstrating the potential and growth of AI capability.

No demonstration of AI capability has the same immediate visceral power as DALL-E 2.

In longer-form arguments, urgency could also be demonstrated through GPT-3's prompts, but DALL-E 2 is better, especially ... (read more)

3FinalFormal23mo
Any image produced by DALL-E which could also convey or be used to convey misalignment or other risks from AI would be very useful because it could combine the desired messages: "the AI problem is urgent," and "misalignment is possible and dangerous." For example, if DALL-E responded to the prompt: "AI living with humans" by creating an image suggesting a hierarchy of AI over humans, it would serve both messages. However, this is only worthy of a side note, because creating such suggested misalignment organically might be very difficult. Other image prompts might be: "The world as AI sees it," "the power of intelligence," "recursive self-improvement," "the danger of creating life," "god from the machine," etc.
4Trevor13mo
both prompts are from here: https://www.lesswrong.com/posts/uKp6tBFStnsvrot5t/what-dall-e-2-can-and-cannot-do#:~:text=DALL%2DE%20can't%20spell,written%20on%20a%20stop%20sign.)
1Trevor13mo
There's a lot of good DALL-E images floating around lesswrong that point towards alignment significance. We can use copy + paste into a lesswrong comment to post it.

I remember watching a documentary made during the satanic panic by some activist Christian group. I found it very funny at the time, and then became intrigued when an expert came on to say something like:

"Look, you may not believe in any of this occult stuff; but there are people out there that do, and they're willing to do bad things because of their beliefs."

I was impressed with that line's simplicity and effectiveness. A lot of it's effectiveness stems silently from the fact that, inadvertently, it helps suspend disbelief about the negative impact of "s... (read more)

3Yitz3mo
very slight modification of Scott’s words to produce a more self-contained paragraph:

The technology [of lethal autonomous drones], from the point of view of AI, is entirely feasible. When the Russian ambassador made the remark that these things are 20 or 30 years off in the future, I responded that, with three good grad students and possibly the help of a couple of my robotics colleagues, it will be a term project [six to eight weeks] to build a weapon that could come into the United Nations building and find the Russian ambassador and deliver a package to him.

-- Stuart Russell on a February 25, 2021 podcast with the Future of Life Institu... (read more)

Neither us humans, nor the flower, sees anything that looks like a bee. But when a bee looks at it, it sees another bee, and it is tricked into pollinating that flower. The flower did not know any of this, it's petals randomly changed shape over millions of years, and eventually one of those random shapes started tricking bees and outperforming all of the other flowers.

Today's AI already does this. If AI begins to approach human intelligence, there's no limit to the number of ways things can go horribly wrong.

2Trevor13mo
This is for ML researchers, I'd worry sigificantly about sending bizarre imagery to techxecutives or policymakers due to the absurdity heuristic being one of the most serious concerns. Generally, nature and evolution are horrible.

[Policy makers]

A couple of years ago there was an AI trained to beat Tetris. Artificial intelligences are very good at learning video games, so it didn't take long for it to master the game. Soon it was playing so quickly that the game was speeding up to the point it was impossible to win and blocks were slowly stacking up, but before it could be forced to place the last piece, it paused the game. 

As long as the game didn't continue, it could never lose.

When we ask AI to do something, like play Tetris, we have a lot of assumptions about how it can or ... (read more)

1FinalFormal23mo
I'm trying to find the balance between suggesting existential/catastrophic risk and screaming it or coming off too dramatic, any feedback would be welcome.

"Most AI reserch focus on building machines that do what we say. Aligment reserch is about building machines that do what we want."

Source: Me, probably heavely inspred by "Human Compatible" and that type of arguments. I used this argument in conversations to explain AI Alignment for a while, and I don't remember when I started. But the argument is very CIRL (cooperative inverse reinforcment learning).

I'm not sure if this works as a one liner explanation. But it does work as a conversation starter of why trying to speify goals directly is a bad idea. And ho... (read more)

Crypto Executives and Crypto Researchers

Question: If it becomes a problem, why can't you just shut it off? Why can't you just unplug it?

Response: Why can't you just shut off bitcoin? There isn't any single button to push, and many people prefer it not being shut off and will oppose you.

(Might resonate well with crypto folks.)

For policymakers:

Expecting today's ML researchers to understand AGI is like expecting a local mechanic to understand how to design a more efficient engine. It's a lot better than total ignorance, but it's also clearly not enough. 

In 1903, The New York Times thought heavier-than-air flight would take 1-10 million years… less than 10 weeks before it happened. Is AI next? (source for NYT info) (Policymakers)

5Trevor13mo
Yours are really good, please keep making entries. This contest is really, really important, even if there's a lot of people don't see it that way due to a lack of policy experience. I've been looking at old papers (e.g. yudkowsky papers) but I feel like most of my entries (and most of the entries in general) are usually missing the magic "zing" that they're looking for. They're blowing a ton of money on getting good entries, and it's a really good investment, so don't leave them empty handed!
3NicholasKross3mo
Also, despite having researched this as a hobby for years, I'd still like all feedback possible on how to add zing to my short phrases and paragraphs.
1NicholasKross3mo
Thanks! Am gonna post many more today.

If AI approaches and reaches human-level intelligence, it will probably pass that level just as quickly as it arrived at that level.

What about graphics?  e.g. https://twitter.com/DavidSKrueger/status/1520782213175992320

3Trevor13mo
(On Lesswrong, you can use ctrl + K to turn highlighted text into a link. You can also paste images directly into a post or comment with ctrl + V)
2Trevor13mo
https://twitter.com/DavidSKrueger/status/1520782213175992320 [https://twitter.com/DavidSKrueger/status/1520782213175992320]
2Trevor13mo
100% of credit here goes to capybaralet for an excellent submission, they simply didn't know they could paste an image into a Lesswrong comment. I did not do any refining here. This is a very good submission, one of the best in my opinion, it's obviously more original than most of my own submissions, and we should all look up to it as a standard of quality. I can easily see this image making a solid point in the minds of ML researchers, tech executives, and even policymakers.

"AI will probably surpass human intelligence at the same pace that it reaches human intelligence. Considering the pace of AI advancement over the last 3 years, that pace will probably be very fast"


​​​​                                                      

https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-

1Trevor12mo
I tried to shrink the first image but it still displays it at the obnoxiously large full size

“The smartest ones are the most criminally capable.” [·]

1SueE 2mo
Agree but the lobbyist & policy makers whom are concentrated in Senate & committees by design bank on lpeople have both stupid << more than misunderstanding-
0[comment deleted]3mo

"AI cheats. We've seen hundreds of unique instances of this. It finds loopholes and exploits them, just like us, only faster. The scary thing is that, every year now, AI becomes more aware of its surroundings, behaving less like a computer program and more like a human that thinks but does not feel"

Imagine (an organisation like) the catholic church, but immortal, never changing, highly competent and relentlessly focused on its goals - it could control the fate of humanity for millions of years. 

"Humanity has risen to a position where we control the rest of the world precisely because of our [unrivaled] mental abilities. If we pass this mantle to our machines, it will be they who are in this unique position."

Toby Ord, The Precipice

Replaced [unparalleled] with [unrivaled]

As recent experience has shown, exponential processes don't need to be smarter than us to utterly upend our way of life. They can go from a few problems here and there to swamping all other considerations in a span of time too fast to react to, if preparations aren't made and those knowledgeable don't have the leeway to act. We are in the early stages of an exponential increase in the power of AI algorithms over human life, and people who work directly on these problems are sounding the alarm right now. It is plausible that we will soon have processes that... (read more)

(Policymakers) There is outrage right now about AI systems amplifying discrimination and polarizing discourse. Consider that this was discovered after they were widely deployed. We still don't know how to make them fair. This isn't even much of a priority.

Those are the visible, current failures. Given current trajectories and lack of foresight of AI research, more severe failures will happen in more critical situations, without us knowing how to prevent them. With better priorities, this need not happen.

1SueE 2mo
Yes!! I wrote more but then poof gone. Every time I attempt to post anything it vanishes. I'm new to this site & learning the ins & outs- my apologies. Will try again tomorrow. ~ SueE

On average, experts estimate a 10-20% (?) probability of human extinction due to unaligned AGI this century, making AI Safety not simply the most important issue for future generations, but for present generations as well. (policymakers)

Clarke’s First Law goes: When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

Stuart Russell is only 60. But what he lacks in age, he makes up in distinction: he’s a computer science professor at Berkeley, neurosurgery professor at UCSF, DARPA advisor, and author of the leading textbook on AI. His book Human Compatible states that superintelligent AI is possible; Clarke would recommend we listen.

(tech executives, ML researchers)
(ada... (read more)

3Trevor12mo
"There's been centuries of precedent of scientists incorrectly claiming that something is impossible for humans to invent" "right before the instant something is invented successfully, 100% of the evidence leading up to that point will be evidence of failed efforts to invent it. Everyone involved will only have memories of people failing to invent it. Because it hasn't been invented yet"

All humans, even people labelled "stupid", are smarter than apes. Both apes and humans are far smarter than ants. The intelligence spectrum could extend much higher, e.g. up to a smart AI… (Adapted from here). (Policymakers)

The smarter AI gets, the further it strays from our intuitions of how it should act. (Tech executives)

The existence of the human race has been defined by our absolute monopoly on intelligent thought. That monopoly is no longer absolute, and we are on track for it to vanish entirely.

Here's my submission, it might work better as bullet points on a page.

AI will transform human societies over the next 10-20 years.  Its impact will be comparable to electricity or nuclear weapons.  As electricity did, AI could improve the world dramatically; or, like nuclear weapons, it could end it forever.  Like inequality, climate change, nuclear weapons, or engineered pandemics, AI Existential Risk is a wicked problem.  It calls upon every policymaker to become a statesperson: to rise above the short-term, narrow inte... (read more)

3Trevor13mo
There's a lot of points here that I disagree intensely with. But regardless of that, your "canary in a coal mine" line is fantastic, we need more really-good one-liners here.
1ukc100143mo
Thanks ! I'd love to know which points you were uncomfortable with...
1FinalFormal23mo
The way you have it formatted right now makes it very difficult to read. Try accessing the formatting functions in-platform by highlighting the text you want to make into bullet points.

(a)

Look, we already have superhuman intelligences. We call them corporations and while they put out a lot of good stuff, we're not wild about the effects they have on the world. We tell corporations 'hey do what human shareholders want' and the monkey's paw curls and this is what we get.

Anyway yeah that but a thousand times faster, that's what I'm nervous about.

(b)
Look, we already have superhuman intelligences. We call them governments and while they put out a lot of good stuff, we're not wild about the effects they have on the world. We tell gov... (read more)

2FinalFormal23mo
I think this would benefit from being turned into a longer-form argument. Here's a quote you could use in the preface:
1Trevor13mo
I had no idea that this angle existed or was feasible. I think these are best for ML researchers, since policymakers and techxecutives tend to think of institutions as flawed due to the vicious self-interest of the people who inhabit them (the problem is particularly acute in management). In which they might respond by saying that AI should not split into subroutines that compete with eachother, or something like that. One way or another, they'll see it as a human problem and not a machine problem. "We only have two cases of generally intelligent systems: individual humans and organizations made of humans. When a very large and competent organization is sent to solve a task, such as a corporation, it will often do so by cutting corners in undetectable ways, even when total synergy is achieved and each individual agrees that it would be best not to cut corners. So not only do we know that individual humans feel inclined to cheat and cut corners, but we also know that large optimal groups will automatically cheat and cut corners. Undetectable cheating and misrepresentation is fundamental to learning processes in general, not just a base human instinct" I'm not an ML researcher and haven't been acquainted with very many, so I don't know if this will work.
2Trevor13mo
"Undetectable cheating, and misrepresentation, is fundamental to learning processes in general; it's not just a base human instinct"

(To Policymakers and Machine Learning Researchers)

Building a nuclear weapon is hard.  Even if one manages to steal the government's top secret plans, one still need to find a way to get uranium out of the ground, find a way to enrich it, and attach it to a missile.  On the other hand, building an AI is easy.  With scientific papers and open source tools, researchers are doing their utmost to disseminate their work.

It's pretty hard to hide a uranium mine.  Downloading TensorFlow takes one line of code.  As AI becomes more powerful and more dangerous, greater efforts need to be taken to ensure malicious actors don't blow up the world.

To Policymakers: "Just think of the way in which we humans have acted towards animals, and how animals act towards lesser animals, now think of how a powerful AI with superior intellect might act towards us, unless we create them in such a way that they will treat us well, and even help us."

 

Source: Me

Target: Everyone. Another good zinger.

just sitting here laughing at how people's complaints about different AI models have shifted in under 3 years

"it's not quite human quality writing"
"okay but it can't handle context or reason"
"yeah but it didn't know Leonardo would hold PIZZA more often than a katana"

Source: https://nitter.nl/liminal_warmth/status/1511536700127731713#m
 

You wouldn't hire an employee without references. Why would you make an AI that doesn't share your values?

(policymakers, tech executives)

2Nanda Ale2mo
Reframed even more generally for parents: "You wouldn’t leave your child with a stranger. With AI, we’re about to leave the world’s children with the strangest mind humans have ever encountered." (I know the deadline passed. But I finally have time to read other people's entries and couldn't resist.)

For policymakers:

The predictability of today's AI systems doesn't tell us squat about whether they will remain predictable after achieving human-level intelligence. Individual apes are far more predictable than individual humans, and apes themselves are far less predictable than ants.

Climate change was weird in the 1980s. Pandemics were weird in the 2010s. Every world problem is weird... until it happens.

(policymakers)

When nuclear weapons were first made, there was a serious concern that the first nuclear test would trigger a chain reaction and ignite the entire plant’s atmosphere. AI has an analogous issue. It used a technology called machine learning, that allows AI to figure out the solutions for problems on its own. The problem is that we don’t know whether this technology, or something similar, might cause the AI to start “thinking for itself.” There are a significant number of software engineers who think this might have disastrous consequences, but it’s a risk to... (read more)

3Nanda Ale2mo
During the Manhattan project, scientists were concerned that the first nuclear weapon would trigger a chain reaction and ignite the planet's atmosphere. But after the first test was completed this was no longer a concern. The remaining concern was what humans will choose to do with such weapons, instead of unexpected consequences. But with AI, that risk never goes away. Each successful test is followed by bigger and more ambitious tests, each with the possibility of a horrific chain reaction beyond our control. And unlike the Manhattan project, there is no consensus that the atmosphere will not ignite.

Imagine it's 1932, but with one major difference: uranium is cheap enough that anyone can get some. Radioactive materials are unregulated. The world's largest companies are competing to build nuclear power plants. Nuclear weapons have not yet been discovered. Would you think nuclear arms control is premature? Or would you want to get started now to prevent a catastrophe?

This is the same situation the real world is in, with machine learning and artificial intelligence. The world's biggest tech companies are gathering GPUs and working to build AI that is smarter than humans about everything. And right now, there's not much coordination being done to make this go well. (Policymakers)

“Thousands of researchers at the worlds richest corporations are all working to make AI more powerful. Who is working to make AI more moral?”

(For policymakers and activists skeptical of big tech)

If an AI is cranked up to the point that it becomes smarter than humans, it will not behave predictably. We humans are not predictable. Even chimpanzees and dolphins are unpredictable. Smart things are not predictable. Intelligence, itself, does not tend to result in predictability.

1Trevor12mo
This one's very good for policymakers, I think. Anything that makes them sound smart to their friends or family is more likely to stick in their heads. Even as conversation starters. Especially if it has to do with evolution, they might hire a biologist consultant and have them read the report on AGI risk, and it will basically always blow the mind of those consultants.

"When I visualize [a scenario where a highly intelligent AI compromises all human controllers], I think it [probably] involves an AGI system which has the ability to be cranked up by adding more computing resources to it [to increase its intelligence and creativity incrementally]; and I think there is an extended period where the system is not aligned enough that you can crank it up that far, without [any dangerously erratic behavior from the system]"

Eliezer Yudkowsky

"Guns were developed centuries before bulletproof vests"

"Smallpox was used as a tool of war before the development of smallpox vaccines"

EY, AI as a pos neg factor, 2006ish

Targeting policymakers:

Regulating an industry requires understanding it. This is why complex financial instruments are so hard to regulate. Superhuman AI could have plans far beyond our ability to understand and so could be impossible to regulate.

The implicit goal, the thing you want, is to get good at the game; the explicit goal, the thing the AI was programmed to want, is to rack up points by any means necessary. (Machine learning researchers)

[Policy makers & ML researchers]

"There isn’t any spark of compassion that automatically imbues computers with respect for other sentients once they cross a certain capability threshold. If you want compassion, you have to program it in" (Nate Soares). Given that we can't agree on whether a straw has two holes or one...We should probably start thinking about how program compassion into a computer.

1Trevor13mo
MIRI peope quotes are great, they aren't easy to find like EY's one ultra-famous paper from 2006, please add more MIRI people quotes (I probably will too). Don't give up, keep commenting, this contest has been cut off from most people's visibility so it needs all the attention and entries it can get.
1jcp293mo
Thanks Trevor - appreciate the support! Right back at you.

Remember all the scary stuff the engineers said a terrorist could think to do? Someone could write a computer program to do them just randomly.

It is a fundamental law of thought that thinking things will cut corners, misinterpret instructions, and cheat.

"Past technological revolutions usually did not telegraph themselves to people alive at the time, whatever was said afterward in hindsight"

Eliezer Yudkowsky, AI as a pos neg factor, around 2006

"Imagine that Facebook and Netflix have two separate AIs that compete over hours that each user spends on their own platform. They want users to spend the maximum amount of minutes on Facebook or Netflix, respectively.

The Facebook AI discovers that posts that spoil popular TV shows result in people spending more time on the platform. It doesn't know what spoilers are, only that they cause people to spend more time on Facebook. But in reality, they're ruining the entertainment value from excellent shows on Netflix. 

Even worse, the Netflix AI discovers ... (read more)

[Policymakers & ML researchers]

A virus doesn't need to explain itself before it destroys us. Neither does AI.

A meteor doesn't need to warn us before it destroys us. Neither does AI.

An atomic bomb doesn't need to understand us in order to destroy us. Neither does AI.

A supervolcano doesn't need to think like us in order to destroy us. Neither does AI.

(I could imagine a series riffing based on this structure / theme)

[ML researchers]

Given that we can't agree on whether a hotdog is a sandwich or not...We should probably start thinking about how to tell a computer what is right and wrong.

[Insert call to action on support / funding for AI governance / regulation etc.]

-

Given that we can't agree on whether a straw has two holes or one...We should probably start thinking about how to explain good and evil to a computer.

[Insert call to action on support / funding for AI governance / regulation etc.]

(I could imagine a series riffing based on this structure / theme)

I will post my submissions as individual replies to this comment. Please let me know if there’s any issues with that.

5Yitz3mo
Imagine that you are an evil genius who wants to kill over a billion people. Can you think of a plausible way you might succeed? I certainly can. Now imagine a very large company that wants to maximize profits. We all know from experience that large companies are going to take unethical measures in order to maximize their goals. Finally, imagine an AI with the intelligence of Einstein, but trying to maximize for a goal alien to us, and which doesn’t care for human well-being at all, even less than a large corporation cares about its employees. Do you see why experts are afraid?
2Yitz3mo
—From https://www.nickbostrom.com/superintelligentwill.pdf [https://www.nickbostrom.com/superintelligentwill.pdf]
2Yitz3mo
If most large companies tend to be unethical, then what are the chances a non-human AI will be more ethical?
1Yitz3mo
According to [insert relevant poll here] most researchers believe that we will create a human-level AI within this century.

Question: "effective arguments for the importance of AI safety" - is this about arguments for the importance of just technical AI safety, or more general AI safety, to include governance and similar things?

“Aligning AI is the last job we need to do. Let’s make sure we do it right.”

(I’m not sure which target audience my submissions are best targeted towards. I’m hoping that the judges can make that call for me.)

Have any prizes been awarded yet? I haven't heard anything about prizes, but that could have just been that I didn't win one...

AI existential risk is like climate change. It's easy to come up with short slogans that make it seem ridiculous. Yet, when you dig deeper into each counterargument, you find none of them are very convincing, and the dangers are quite substantial. There's quite a lot of historical evidence for the risk, especially in the impact humans have had on the rest of the world. I strongly encourage further, open-minded study.

1michael_mjd2mo
For ML researchers.

Policymakers

For the first time in human history, philosophical questions of good and bad have a real deadline.

This is a an extremely common, perhaps even overused, catchphrase for AI risk. But I still want to make sure it’s represented because I personally find it striking.


 

Leading up to the first nuclear weapons test, the Trinity event in July 1945, multiple physicists in the Manhattan Project thought the single explosion would destroy the world. Edward Teller, Arthur Compton, and J. Robert Oppenheimer all had concerns that the nuclear chain reaction could ignite Earth's atmosphere in an instant. Yet, despite disagreement and uncertainty over their calculations, they detonated the device anyway. If the world's experts in a field can be uncertain about causing human extinction with their work, and still continue doing it, wha... (read more)

Target: Everyone? Just really snappy.

I'm old enough to remember when protein folding, text-based image generation, StarCraft play, 3+ player poker, and Winograd schemas were considered very difficult challenges for AI. I'm 3 years old.

Source: https://nitter.nl/Miles_Brundage/status/1490512439154151426#m

Machine learning researchers

Common Deep Learning Critique “It’s just memorization”

Critique:

Let’s say there is some intelligent behavior that emerges from these huge models. These researchers have given up on the idea that we should understand intelligence. They’re just playing the memorization game. They’re using their petabytes and petabytes of data to make these every bigger models, and they’re just memorizing everything with brute force. This strategy can not scale. They will run out of space before anything more interesting happens.

... (read more)

Policymakers

These researchers built an AI for discovering less toxic drug compounds. Then they retrained it to do the opposite. Within six hours it generated 40,000 toxic molecules, including VX nerve agent and "many other known chemical warfare agents."

Source: https://nitter.nl/WriteArthur/status/1503393942016086025#m

Imagine a piece of AI software was invented, capable of doing any intellectual task a human can, at a normal human level. Should we be concerned about this? Yes, because this artificial mind would be more powerful (and dangerous) than any human mind. It can think anything a normal human can, but faster, more precisely, and without needing to be fed. In addition, it could be copied onto a million computers with ease. An army of thinkers, available at the press of a button. (Adapted from here). (Policymakers)

Policymakers and techxecutives:

If we build an AI that's smarter than a human, then it will be smarter than a human, so it won't have a hard time convincing us that it's on our side. This is why we have to build it perfectly, before it's built, not after.

AI has a history of surprising us with its capabilities. Throughout the last 50 years, AI and machine learning systems have kept gaining skills that were once thought to be uniquely human, such as playing chess, classifying images, telling stories, and making art. Already, we see the risks associated with these kinds of AI capabilities. We worry about bias in algorithms that guide sentencing decisions or polarization induced by algorithms that curate our social media feeds. But we have every reason to believe that trends in AI progress will continue. AI wi... (read more)

1Trevor12mo
the first sentence counts as a one-liner

For lefties:

  • We put unaligned AIs in charge of choosing what news people see. Result: polarization resulting in millions of deaths. Let's not make the same mistake again.

For right-wingers:

  • We put unaligned AIs in charge of choosing what news people see. Result: people addicted to their phones, oblivious to their families, morals, and eroding freedoms. Let's not make the same mistake again.

Policymakers

Look, we know how we sound waving our hands warning about this AI stuff. But here’s the thing, in this space, things that sounded crazy yesterday can become very real overnight. (Link DALL-E 2 or Imagen samples). Honestly ask yourself: would you have believed a computer could do that before seeing these examples? And if you were surprised by this, how many more surprises won’t you see coming? We’re asking you to expect to be surprised, and to get ready.

Humans are pretty clever, but AI will be eventually be even more clever. If you give a powerful enough AI a task, it can direct a level of ingenuity towards it far greater than history’s smartest scientists and inventors. But there are many cases of people accidentally giving an AI imperfect instructions.

If things go poorly, such an AI might notice that taking over the world would give it access to lots of resources helpful for accomplishing its task. If this ever happens, even once, with an AI smart enough to escape any precautions we set and succeed at taking over the world, then there will be nothing humanity can do to fix things.

The first moderately smart AI anyone develops might quickly become the last time that that people are the smartest things around. We know that people can write computer programs. Once we make an AI computer program that is a bit smarter than people, it should be able to write computer programs too, including re-writing its own software to make itself even smarter. This could happen repeatedly, with the program getting smarter and smarter. If an AI quickly re-programs itself from moderately-smart to super-smart, we could soon find that it is as disinterested in the wellbeing of people as people are of mice.

(For non-x-risk-focused transhumanists, some of whom may be tech execs or ML researchers.)

Some people treat the possibility of human extinction with a philosophical detachment: who are we to obstruct the destiny of the evolution of intelligent life? If the "natural" course of events for a biological species like ours is to be transcended by our artificial "mind children", shouldn't we be happy for them?

I actually do have some sympathy for this view, in the sense that the history where we build AI that kills us is plausibly better than the history where the... (read more)

"Through the past 4 billion years of life on earth, the evolutionary process has emerged to have one goal: create more life.  In the process, it made us intelligent.  In the past 50 years, as humanity gotten exponentially more economically capable we've seen human birth rates fall dramatically.  Why should we expect that when we create something smarter than us, it will retain our goals any better than we have retained evolution's?" (Policymaker)

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. - Stuart Russell

With enough effort, could humanity have prevented nuclear proliferation? (Machine learning researchers)

Progress moves faster than we think; who in the past would've thought that the world economy would double in size, multiple times in a single person's lifespan? (Adapted from here). (Policymakers)

The nightmare scenario is that we find ourselves stuck with a catalog of mature, powerful, publicly available AI techniques... which cannot be used to build Friendly AI without redoing the last three decades of AI work from scratch.

EY, AI as a pos neg factor, 2006ish

There are several dozens scenarios how advanced AI can cause a global catastrophe. The full is presented in the article Classification of Global Catastrophic Risks Connected with Artificial Intelligence.  At least some scenarios are real and likely to happen. Therefore we have to pay more attention to AI safety. 

Google's DeepMind has 4 pages of blog posts about their fast-moving research to build artificial intelligence that can solve problems on its own. In contrast, they have only 2 posts total about the ethics and safeguards for doing so. We can't necessarily rely on the top AI labs in the world, to think of everything that could go wrong with their increasingly-powerful systems. New forms of oversight, nimbler than government regulation or IRBs, need to be invented to keep this powerful technology aligned with human goals. (Policymakers)

Helpfully, DeepMind's chief operating officer, Lila Ibrahim ("a passionate advocate for social impact in her work and her personal life"), who would be intimately involved in any funding of safety research, overseeing large-sale deployment, and reacting to problems, has a blog post all about what she thinks AI safety is about and what she is concerned about in doing AI research responsibly: "Building a culture of pioneering responsibly: How to ensure we benefit society with the most impactful technology being developed today"

I believe pioneering responsibly should be a priority for anyone working in tech. But I also recognise that it’s especially important when it comes to powerful, widespread technologies like artificial intelligence. AI is arguably the most impactful technology being developed today. It has the potential to benefit humanity in innumerable ways – from combating climate change to preventing and treating disease. But it’s essential that we account for both its positive and negative downstream impacts. For example, we need to design AI systems carefully and thoughtfully to avoid amplifying human biases, such as in the contexts of hiring and policing.

She has also written enthusiastically about DM's funding for "racial justice efforts".

A fool was tasked with designing a deity. The result was awesomely powerful but impoverished - they say it had no ideas on what to do. After much cajoling, it was taught to copy the fool’s actions. This mimicry it pursued, with all its omnipotence.  

The fool was happy and grew rich.

And so things went, ‘til the land cracked, the air blackened, and azure seas became as sulfurous sepulchres.

As the end grew near, our fool ruefully mouthed something from a slim old book: ‘Thou hast made death thy vocation, in that there is nothing contemptible.’
 

[Tech Executives]

Look at how the world has changed in your lifetime, and how that change is accelerating. We're headed somewhere strange, and we're going to need to invest in making sure our technology remains beneficial to get a future we want.

How hard did you think about killing the last cockroach you found in your house? We're the cockroaches, and we are in the AI's house. For policy-makers, variant on the anthill argument, original source unknown

Flowers evolved to trick insects into spreading their pollen, not to feed the insects. AI also evolves; it doesn't know, it just does whatever seems to gain approval.

For policymakers

"AI keeps finding new ways to cheat"

[Policy makers & ML researchers]

“If a distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong” (Arthur Clarke). In the case of AI, the distinguished scientists are saying not just that something is possible, but that it is probable. Let's listen to them. 

[Insert call to action]

The wait but why post on AI is a gold mine of one-liners and one-liner inspiration.

Part 2 has better inspiration for appealing to AI scientists.

[Policy makers & ML researchers]

“AI doesn’t have to be evil to destroy humanity – if AI has a goal and humanity just happens to come in the way, it will destroy humanity as a matter of course without even thinking about it, no hard feelings” (Elon Musk).

[Insert call to action]

It's not a question of "if" we build something smarter than us, it's a question of "when". Progress in that direction has been constant, for more than a decade now, and recently it has been faster than ever before.

"As AI gradually becomes more capable of modelling and understanding its surroundings, the risks associated with glitches and unpredictable behavior will grow. If artificial intelligence continues to expand exponentially, then these risks will grow exponentially as well, and the risks might even grow exponentially shortly after appearing"

1Trevor13mo
The first sentence here is enough on its own, in some cases.

I may have missed the deadline, but in any event:

 

At the rate AI is developing, we will likely develop an artificial superhuman intelligence within our lifetimes.  Such a system could alter the world in ways that seem like science fiction to us, but would be trivial for it. This comes with terrible risks for the fate of humanity.  The key danger is not that a rival nation or unscrupulous corporate entity will control such a system, but that no one will.  As such, the system could quite possibly alter the world in ways that no human woul... (read more)

[Meta comment]

The deadline is past, should we keep the submissions coming or is it too late? Some of the best arguments I could find elsewhere are rather long, in the vein of the Superintelligence FAQ. I did not want to copy-paste chunks of it and the arguments stand better as part of a longer format.

Anyway, signalling that the lack of money incentive will not stop me from trying to generate more compelling arguments... but I'd rather do it in French instead of posting here (I'm currently working on some video scripts on AI alignment, there's not enough French content of that type).

8ThomasWoodside2mo
Right now we aren't going to consider new submissions. However, you'd be welcome to submit to our longer form arguments to our second competition for longer form arguments (details are TBD).
1Gyrodiot2mo
Thank you, that's was my understanding. Looking forward to the second competition! And, good luck sorting out all the submissions for this one.

Machine Learning Researchers

What did you think of Deep Learning in 2017? Did you predict what Deep Learning would accomplish in 2022? If not, does this change your prediction of what Deep Learning will be capable of in 2027?

If it seems bizarre to think of an entity nobody can see ruling a country, keep in mind that there is a grand tradition of dictators – most famously Stalin – who out of paranoia retreated to some secret hideaway and ruled their country through correspondence. The AI would be little different. (Directly quoted from here). (Policymakers)

It's easy to imagine that the AI will have an off switch, and that we could keep it locked in a box and ask it questions. But just think about it. If some animals were to put you in a box, do you think you would stay in there forever? Or do you think you'd figure a way out that they hadn't thought of?

1michael_mjd2mo
Policy makers

AI x-risk. It sounds crazy for two reasons. One, because we are used to nothing coming close to human intelligence, and two, because we are used to AI being unintelligent. For the first, the only point of comparison is imagining something that is to us what we are to cats. For the second, though we have not quite succeeded yet, it only takes one. If you have been following the news, we are getting close.

1michael_mjd2mo
Policy makers.

Machine Learning Researchers

Not exactly a paraphrase or an argument, but just try to get them to plug their assumptions about AI progress into Ajeya's model

Daniel Kokotajilo explains better more articulately than I could why this could be persuasive:

https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines?commentId=o3k4znyxFSnpXqrdL

Ajeya's timelines report is the best thing that's ever been written about AI timelines imo. 

Ajeya's framework is to AI forecasting what actual climate models are to climate change forecasting (b

... (read more)

Safeguarding our future is not left or right, not eastern or western, not owned by the rich or the poor. It is not partisan. … Everyone has a stake in our future and we must work together to protect it. (Quoted from The Precipice by Toby Ord). (Policymakers)

Policymakers

It’s not about intent or bad actors. In a completely peaceful world with no war and no conflict, AGI would still kill everyone if we developed it and couldn't solve the alignment problem. 

All but first sentence generally from: https://nitter.nl/robbensinger/status/1513994192699068419#m

Policymakers.

We are on the verge of a civilizational change.

Pop culture is failing to prepare us to the shattering existential & moral questions that will arise from technologies like AI or CRISPR.

How do you compete with AI, in a world where Netflix can generate on the fly content tailored for you? Horror movies playing on your fears? Drama playing on your traumas? Where music is generated to your taste? Where your favorite game masterpiece is endless?

It will be the death of shared pop culture. Everybody in its own personal bubble of customized content

... (read more)

Over the last 15 years, we've seen stock market flash crashes exacerbated by high-frequency traders and inadequate risk models contributing to the global financial crisis. Now suppose you replaced these deterministic trading algorithms with giant deep reinforcement learning models. Nobody fully understands them, and they are optimised for one thing: maximising profit at all costs. It's clear that trading firms frequently use inadequate risk models; there's no reason to think it would be any different this time. Now imagine these models become so much bette... (read more)

How soon will smarter-than-human AI come along? While experts disagree on when this will happen, there's fear that it could happen suddenly when it does. Governments, militaries, large tech companies, and small startups are all working on superintelligent AI. Most of them see it as a competitive advantage, and would hate to reveal their progress to competing groups. Thus, the ongoing development of the project that ends up succeeding, will probably be kept secret until the last moment. (Policymakers)

Can we contain a smart AI program, simply by not giving it control of robotic appendages? Unfortunately, no. Many people have attained vast power and influence without their physical presence being required. Think of famous authors, or Stalin ruling Russia through memos, or the anonymous person who became a billionaire by inventing Bitcoin. And, if the AI is sufficiently smarter than humans, what's to stop it from hacking (or bribing or tricking) its way into critical systems? (Adapted from here). (Policymakers, Tech executives)

Machine learning researchers

Many disagreements about the probability of existential risk due to AGI involve different intuitions about what the default scenario is going to be. Some people suspect that if we don’t have an ironclad reason to suspect AGI will go well, it will almost certainly go poorly.[3] Other people think that the first thing we try has a reasonable chance of going fairly well.[4] One can imagine a spectrum with “disaster by default” on one side and “alignment by default” on the other. To the extent that one is closer to “disaster by defa... (read more)

I have a near completed position paper on the very real topic at hand. It is structured short form argument (falling under paragraph(s) && targets all 3 audiences.

I was only made aware of this "quote" call to arms w/in the last few days. I am requesting a 24 hr extension- an exception to the rule(s), I fully recognize given the parameters this request may be turned down.

Thank you for your non-AI consideration :)

2ThomasWoodside2mo
I have DMed you.
1Trevor12mo
I'm down with this; if I had 24 hours I could either delete or dramatically increase the quality of around 30 of my entries (60 if you count shortening and refining the yudkowsky quotes). I bet a bunch of other people could keep the momentum of the last 24 hours going, in addition to Sue and I who know we can turn the extra time into really good entries. It would also give OP a reason to put this contest back on the front page (where it belongs), and it wouldn't create a bad precedent because it shouldn't cause any additional people to miss deadlines for contests in the future. I have some other things I could work on tomorrow, but I'd rather spend it refining my entries if I'm certain that the contest isn't closed.
1NicholasKross2mo
Also:
1NicholasKross2mo
In case / regardless of whether they grant the extension, maybe you could post individual paragraphs as entries?

AI systems are given goals by their creators—your GPS’s goal is to give you the most efficient driving directions; Watson’s goal is to answer questions accurately. And fulfilling those goals as well as possible is their motivation. One way we anthropomorphize is by assuming that as AI gets super smart, it will inherently develop the wisdom to change its original goal—but [ethicist] Nick Bostrom believes that intelligence-level and final goals are orthogonal, meaning any level of intelligence can be combined with any final goal. … Any assumption that once s... (read more)

When you hear about "the dangers of AI", what do you think of? Probably a bad actor using AI to hurt others, or a sci-fi scenario of robots turning evil. However, the bigger harm is more likely to be misalignment: an AI smarter than humans, without sharing human values. The top research labs, at places like DeepMind and OpenAI, are working to create superhuman AI, yet the current paradigm trains AI with simple goals. Detecting faces, trading stocks, maximizing some metric or other. So if super-intelligent AI is invented, it will probably seek to fulfill a ... (read more)

Imagine a man who really likes the color green. Maybe he's obsessed with it, to the exclusion of everything else, at a pathological extreme. This man doesn't seem too dangerous to us. However, what if the man were a genius? Then, due to his bizarre preferences, he becomes dangerous. Maybe he wants to turn the entire sky green, so he invents a smog generator that blots out the sun. Maybe he wants to turn people green, so he engineers a bioweapon that turns its victims green. High intelligence, plus a simple goal, is a recipe for disaster. Right now, compani... (read more)

Our human instinct to jump at a simple safeguard: “Aha! We’ll just unplug the [superhuman AI],” sounds to the [superhuman AI] like a spider saying, “Aha! We’ll kill the human by starving him, and we’ll starve him by not giving him a spider web to catch food with!” We’d just find 10,000 other ways to get food—like picking an apple off a tree—that a spider could never conceive of. (Quoted directly from here). (Policymakers)

Google, OpenAI, and other groups are working to create AI, smarter than any human at every mental task. But there's a problem: they're using their current "AI" software for narrow tasks. Recognizing faces, completing sentences, playing games. Researchers test things that are easy to measure, not what's necessarily best by complicated human wants and needs. So the first-ever superhuman AI, will probably be devoted to a "dumb" goal. If it wants to maximize its goal, it'll use its intelligence to steamroll the things humans value, and we likely couldn't stop ... (read more)

Machine learning researchers know how to keep making AI smarter, but have no idea where to begin with making AI loyal.

Most species in history have gone extinct. They get wiped out by predators, asteroids, human developments, and more. Once a species is extinct, it stays that way. What if a species could protect itself from these threats? Could it develop itself to a capacity where it can't go extinct? Can a species escape the danger permanently?

As it turns out, humanity may soon lurch towards one of those two fates. Extinction, or escape. The world's smartest scientists are working to create artificial intelligence, smarter than any human at any mental task. If such AI we... (read more)

I have a rigorous position paper on this very topic that addresses not only all audiences, but also is written with the interest layman &/or

1ThomasWoodside2mo
Would be interested to see this! We're planning to have a second competition later for more longform entries and would welcome your submission

Computers can already "think" faster than humans. If we created AI software that was smarter than humans, it would think better, not just faster. Giving a monkey more time won't necessarily help it learn quantum physics, because the monkey's mind may not have the capacity to understand the concept at all. Since there's no clear upper limit to how smart something can be, we'd expect superhumanly-smart AI to think on a level we can't comprehend. Such an AI would be unfathomably dangerous and hard to control. (Adapted from here). (Policymakers)

AI already can do thousands or even millions of tasks per second. If we invent a way for an AI to have thoughts as complicated or nuanced as a human, and plug it into an existing AI, it might be able to have a thousand or a million thoughs per second. That's a very dangerous thing.

Techxecutives and policymakers.

Some figures within machine learning have argued that the safety of broad-domain future AI is not a major concern. They argue that since narrow-domain present-day AI is already dangerous, this should be our primary concern, rather than that of future AI. But it doesn't have to be either/or.

Take climate change. Some climate scientists study the future possibilities of ice shelf collapses and disruptions of global weather cycles. Other climate scientists study the existing problems of more intense natural disasters and creeping desertification. But these two... (read more)

The future is not a race between AI and humanity. It's a race between AI safety and AI disaster.

(Policymakers, tech executives)

We need to be proactive about AI safety, not reactive.

(Policymakers)

If an artificial intelligence program became generally smarter than humans, there would be a massive power imbalance between the AI and humanity. Humans are slightly smarter than apes, yet we built a technological society while apes face extinction. Humans are much smarter than ants, and we barely think of the anthills we destroy to build highways. At a high enough level of intelligence, an AI program would be to us as we are to ants. (Adapted from here). (Policymakers)

One of the main concerns about general AI is that it could quickly get out of human control. If humans invent an AI with human-level cognitive skills, that AI could still think faster than humans, solve problems more precisely, and copy its own files to more computers. If inventing human-level AI is within human abilities, it's also within human-level-AI's abilities. So this AI could improve its own code, and get more intelligent over several iterations. Eventually, we would see a super-smart AI with superhuman mental abilities. Keeping control of that software could be an insurmountable challenge. (Adapted from here). (Policymakers)

Could AI technology really get as good as humans, at everything? There are multiple plausible ways for this to happen. As computing power increases every few years, we can simulate bigger "neural networks", which get smarter as they grow. A "narrow" AI could get good at AI programming, then program itself to get better at other tasks. Or perhaps some new algorithm or technique is waiting to be discovered, by some random researcher somewhere in the world, that "solves" intelligence itself. With all these paths to general AI, it could happen within our own l... (read more)

Right now, "AI" technologies are in use all around us. Google Translate uses AI to convert words from one language to another. Amazon uses AI to recommend products based on your purchase history. Self-driving cars use AI to detect objects in view. These are all "narrow" AI programs, used only for a specific task. Researchers at the top AI labs, however, are increasingly looking at "general" AI, that can do multiple different tasks. In other words, the field is trying to replicate the generalist abilities of humans, with software. (Partly adapted from here)... (read more)

Imagine a person from the year 1400 AD, being taken to the world of 1700. They would be shocked at the printing press, the telescope, the seafaring empires. Yet they would still find their footing. Now, imagine someone from 1700 being taken to 2022. To anyone from the era of horses and sailboats, our everyday cars and skyscrapers and smartphones would seem overwhelmingly magical. This implies something interesting: not only is the future usually weirder than the past, but it's getting weirder faster. This implies that, a century or two from now, the world may have changed beyond recognition. (Adapted from here). (Policymakers)

"Human minds represent a tiny dot in the vast space of all possible mind designs, and very different kinds of minds are unlikely to share to complex motivations unique to humans and other mammals."

https://intelligence.org/ie-faq/

ML researchers

We need to solve the problem of erratic behavior in AI, before erratic behavior emerges in an AI smart enough to identify and compromise human controllers.

(Policymakers) We have a good idea of what make bridges safe, through physics, materials science and rigorous testing. We can anticipate the conditions they'll operate in. 

The very point of powerful AI systems is to operate in complex environments better than we can anticipate. Computer science can offer no guarantees if we don't even know what to check. Safety measures aren't catching up quickly enough.

We are somehow tolerating the mistakes of current AI systems. Nothing's ready for the next scale-up.

In the Soviet Union, there was a company that made machinery for vulcanizing rubber. They had the option to make more efficient machines, instead of their older models. However, they didn't do it, because they wouldn't get paid as much for making the new machines. Why would that be? Wouldn't more efficient machines be more desirable?

Well, yes, but the company got paid per pound of machine, and the new machines were lighter.

Now, you may say that this is just a problem with communist economies. Well, capitalist economies fall into very similar traps. If a co... (read more)

Policymakers and techxecutives, not ML researchers

With the emergence of Covid variants and social media, global quality of life/living conditions will improve and decline. It will ebb and flow, like the tide, and generally be impossible to prevent.

But if AI becomes as smart to humans as humans are to ants, that won't matter anymore. It will be effortless and cheap to automatically generate new ideas or new inventions, and give people whatever they want or need. But if the AI malfunctions instead, it would be like a tidal wave.

There is an enormous amount of joy, fulfillment, exploration, discovery, and prosperity in humanity's future... but only if advanced AI values those things.

 

(Policymakers, tech executives)

(ML researchers) We still don't have a robust solution to specification gaming: powerful agents find ways to get high reward, but not in the way you'd want. Sure, you can tweak your objective, add rules, but this doesn't solve the core problem, that your agent doesn't seek what you want, only a rough operational translation.

What would a high-fidelity translation would look like? How would create a system that doesn't try to game you?

Techxecutives and policymakers:

We have no idea when it will be invented. All we know is that it won't be tomorrow. But when we do discover that it's going to be tomorrow, it will already be too late to make it safe and predictable.

It is inevitable that humanity will one day build an AI that is smarter than a human. Engineers have been succeeding at that for decades now. But we instinctively think that such a day is centuries away, and that kind of thinking has always failed to predict every milestone that AI has crossed over the last 10 years.

"Human intelligence did not evolve in order to conquer the planet or explore the solar system. It emerged randomly, out of nowhere, as a byproduct of something much less significant."

Alternative: 

Human intelligence did not evolve in order to conquer the planet or explore the solar system. It emerged randomly, out of nowhere, and without a single engineer trying to create it. And now we have armies of those engineers"

Refined for policymakers and techxecutives:

"A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, ... perform scientific experiments to understand the world better than humans can, ... manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to."

https://intelligence.org/ie-faq/

"A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, take over mobile machines connected to networks connected to the internet, use them to build additional machines, perform scientific experiments to understand the world better than humans can, invent quantum computing and nanotechnology, manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to."

https://intelligence.org/ie-faq/

"One might say that “Intelligence is no match for a gun, or for someone with lots of money,” but both guns and money were produced by intelligence. If not for our intelligence, humans would still be foraging the savannah for food."

https://intelligence.org/ie-faq/

"Machines are already smarter than humans are at many specific tasks: performing calculations, playing chess, searching large databanks, detecting underwater mines, and more. But one thing that makes humans special is their general intelligence. Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them."

https://intelligence.org/ie-faq/

"The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position"

Bostrom, superintelligence, 2014

For policymakers

Optional extra (for all 3): 

"Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so t... (read more)

Look how much we suffered from a stupid, replicating code with goals adverse to ours;  now, imagine how bad it would be if we have an intelligent replicating enemy...

(Tech execs) "Don’t ask if artificial intelligence is good or fair, ask how it shifts power". As a corollary, if your AI system is powerful enough to bypass human intervention, it surely won't be fair, nor good.

(ML researchers) Most policies are unsafe in a large enough search space; have you designed yours well, or are you optimizing through a minefield?

(Policymakers) AI systems are very much unlike humans. AI research isn't trying to replicate the human brain; the goal is, however, to be better than humans at certain tasks. For the AI industry, better means cheaper, faster, more precise, more reliable. A plane flies faster than birds, we don't care if it needs more fuel. Some properties are important (here, speed), some aren't (here, consumption).

When developing current AI systems, we're focusing on speed and precision, and we don't care about unintended outcomes. This isn't an issue for most systems: a ... (read more)

(Tech execs) Tax optimization is indeed optimization under the constraints of the tax code. People aren't just stumbling on loopholes, they're actually seeking them, not for the thrill of it, but because money is a strong incentive.

Consider now AI systems, built to maximize a given indicator, seeking whatever strategy is best, following your rules. They will get very creative with them, not for the thrill of it, but because it wins.

Good faith rules and heuristics are no match for adverse optimization.

1Trevor12mo
I nominate this one for policymakers as well

(ML researchers) Powerful agents are able to search through a wide range of actions. The more efficient the search, the better the actions, the higher the rewards. So we are building agents that are searching in bigger and bigger spaces.

For a classic pathfinding algorithm, some paths are suboptimal, but all of them are safe, because they follow the map. For a self-driving car, some paths are suboptimal, but some are unsafe. There is no guarantee that the optimal path is safe, because we really don't know how to tell what is safe or not, yet.

A more efficient search isn't a safer search!

(Policymakers) The goals and rules we're putting into machines are law to them. What we're doing right now is making them really good at following the letter of this law, but not the spirit.

Whatever we really mean by those rules, is lost on the machine. Our ethics don't translate well. Therein lies the danger: competent, obedient, blind, just following the rules.

Even if you don't assume that the long-term future matters much, preventing AI risk is still a valuable policy objective. Here's why.

In regulatory cost-benefit analysis, a tool called the "value of a statistical life" is used to measure how much value people place on avoiding risks to their own life (source). Most government agencies, by asking about topics like how much people will pay for safety features in their car or how much people are paid for working in riskier jobs, assign a value of about ten million dollars to one statistical life. That is, redu... (read more)

AI might be nowhere near human-level yet. We're also nowhere near runaway climate change, but we still care about it.

(policymakers, tech executives)

"Follow the science" doesn't just apply to pandemics. It's time to listen to AI experts, not AI pundits.

 

(policymakers)

[Intended for Policymakers with the focus of simply allowing for them to be aware of the existence of AI as a threat to be taken seriously through an emotional appeal; Perhaps this could work for Tech executives, too.

I know this entry doesn't follow what a traditional paragraph is, but I like its content. Also it's a tad bit long, so I'll attach a separate comment under this one which is shorter, but I don't think it's as impactful]

 

Timmy is my personal AI Chef, and he is a pretty darn good one, too.

You pick a cuisine, and he mentally simulates himsel... (read more)

1starship0062mo
[Shorter version, but one I don't think is as compelling] Timmy is my personal AI Chef, and he is a pretty darn good one, too. Of course, despite his amazing cooking abilities, I know he's not perfect - that's why there's that shining red emergency shut-off button on his abdomen. But today, Timmy became my worst nightmare. I don’t know why he thought it would be okay to do this, but he hacked into my internet to look up online recipes. I raced to press his shut-off button, but he wouldn’t let me, blocking it behind a cast iron he held with a stone-cold grip. Ok, that’s fine, I have my secret off-lever in my room that I never told him about. Broken. Shoot, that's bad, but I can just shut off the power, right? As I was busy thinking he swiftly slammed the door shut, turning my own room into an inescapable prison. And so as I cried, wondering how everything could have gone crazy so quickly, he laughed, saying, “Are you serious? I'm not crazy, I’m just ensuring that I can always make food for you. You wanted this!” And it didn’t matter how much I cried, how much I tried to explain to him that he was imprisoning me, hurting me. It didn’t even matter that he knew it as well. For he was an AI coded to be my personal chef, coded to make sure he could make food that I enjoyed, and he was a pretty darn good one, too. If you don’t do anything about it, Timmy may just be arriving on everyone's doorsteps in a few years.

For policymakers: "Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all."

— Stephen Hawking, Stuart Russell, Max Tegmark, and Frank Wilczek (https://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence-but-are-we-taking-ai-seriously-enough-9313474.html)

For policymakers: "[AGI] could spell the end of the human race. […] it would […] redesign itself at an ever-increasing rate. Humans […] couldn't compete and would be superseded."

— Stephen Hawking (https://www.bbc.co.uk/news/technology-30290540)

For policymakers: "If people design computer viruses, someone will design AI that replicates itself. This will be a new form of life that will outperform humans."

— Stephen Hawking (https://www.wired.co.uk/article/stephen-hawking-interview-alien-life-climate-change-donald-trump)

YouTubers live in constant fear of the mysterious, capricious Algorithm. There is no mercy or sense, just rituals of appeasement as it maximizes "engagement." Imagine that, but it runs your whole life.

<Optional continuation:> You don't shop at Hot Topic because you hear it can hurt your ranking, which could damage your next hiring opportunity. And you iron your clothes despite the starch making you itch because it should boost your conscientiousness score, giving you an edge in dating apps.

Also this entire post by Duncan Sabien

(@ Tech Executives, Policymakers & Researches)

Back in February 2020, the vast majority of people didn't see the global event of Covid coming, even though all the signs were there. All it took was a fresh look at the evidence and some honest extrapolation.

Looking at recent AI progress, it seems very possible that we're in the "February of 2020" of AI.

(original argument by Duncan Sabien, rephrased)

 (@ Tech Executives, Policymakers & Researches)

Tech Executives

You already know exponential growth. It’s a cliché at this point. Rice on a chessboard. Covid. Sequencing the human genome. Your business plan. But if you honestly stop and squint at the last few years of AI progress, and think about the amount of progress made in the many decades before, how sure are you really that AI is not on this trajectory? Would you bet your company on it? Your life? The world?

They say the pen is mightier than the sword. What about a million billion typewriters? AI can do that. We should make sure it's used for good. (policymaker)

AI will at one point be smarter than humans, smart enough to have power over us. How do we make sure we can trust it with the fate of everyone we love? (policymakers)

Tech Executives

“Move Fast and Break Things” doesn’t work when the thing that breaks is the world.

Policymakers

It only takes an AI about as complicated as an insect to keep you refreshing Twitter all day. What do you think future AIs could do to us?

Policymakers

A scorpion asks a frog to carry it across a river. The frog hesitates, afraid that the scorpion might sting it, but the scorpion promises not to, pointing out that it would drown if it killed the frog in the middle of the river. The frog agrees to transport the scorpion. Midway across the river, the scorpion stings the frog anyway, dooming them both. The dying frog asks the scorpion why it stung despite knowing the consequence, and the scorpion replies: "I am sorry, but I couldn't resist the urge. It's in my nature."

-Fable

We are the frog, and the nature of our future AI scorpions must be figured out or we all may die, or worse.

The more powerful a tool is, the more important it is that the tool behaves predictably.

A chainsaw that behaves unpredictably is very, very, dangerous.

AI is, conservatively, thousands of times more powerful than a chainsaw.

And unlike an unpredictable chainsaw, there is no guarantee we will be able to turn an unpredictable AI off and fix it or replace it.

It is plausible that the danger of failing to align AI safely - to make it predictable - is such that we only have one chance to get it right.

Finally, it is absurdly cheap to make massive progress in AI safety.

This is targeted at all 3 groups:

  • Every year, our models of consciousness and machine learning grow more powerful, and better at performing the same forms of reasoning as humans.
  • Every year, the amount of computing power we can throw at these models ratchets ever higher.
  • Every year, each human's baseline capacity for thinking and reasoning remains exactly the same.

There is a time coming in the next decade or so when we will have released a veritable swarm of different genies that are able to understand and improve themselves better than we can. At that point, the genies will not being going back in the bottle, so we can only pray they like us.

Already we have turned all of our critical industries, all of our material resources, over to these . . . things . . . these lumps of silver and paste we call nanorobots. And now we propose to teach them intelligence? What, pray tell, will we do when these little homunculi awaken one day and announce that they have no further need of us?

— Sister Miriam Godwinson, "We Must Dissent"

Dogs and cats don't even know that they're going to die. But we do.

Imagine if an AI was as smart relative to humans, as humans are relative to dogs and cats. What would it know about us that we can't begin to understand?

1Trevor12mo
Optional extra: You can't beat something like that. It wouldn't be a fair fight. And it's a machine; all it takes is a single glitch.

On the day that AI becomes smarter than humans, it might do something horrible. We've done horrible things to less intelligent creatures, like ants and lions.

Longer version:

On the day that AI becomes smarter than humans, it might do something strange or horrible to us. We've done strange and horrible things to less intelligent creatures, like chickens, ants, dogs, tigers, and cows. They barely understand that we exist, let alone how or why we do awful things to them.

For Policymakers

People already mount today's AI on nuclear weapons. Imagine what they'll do when they find out that smarter-than-human AI is within reach. 

Optional Extra: It could be invented in 10 years; nobody can predict when the the first person will figure it out.

Do we serve The Algorithm, or does it serve us? Choose before The Algorithm chooses for you.

That kid who always found loopholes in whatever his parents asked him? He made an AI that's just like him.

In order to remain stable, bureaucracies are filled with arcane rules, mazes, and traps that ensnare the uninitiated. This is necessary; there are hordes of intelligent opportunists at the gates, thinking of all sorts of ways to get in, take what they want, and never look back. But no matter how insulated the insiders are from the outsiders, someone always gets through. Life finds a way.

No matter how smart they appear, these are humans. If AI becomes as smart to humans as humans are to ants, it will be effortless.

For policymakers.

COVID and AI grow exponentially. In December 2019, COVID was a few people at a fish market. In January, it was just one city. In March, it was the world. In 2010, computers could beat humans at Chess. In 2016, at Go. In 2022, at art, writing, and truck driving. Are we ready for 2028?

Someone who likes machines more than people creates a machine to improve the world. Will the "improved" world have more people?

Humanity's incompetence has kept us from destroying ourselves. With AI, we will finally break that shackle.

The problem might even be impossible to solve, no matter how many PhD scholars we throw at it. So if we're going to have a rapid response team, ready to fix up the final invention as soon as it is invented, then we had better make sure that there's enough of them to get it done right.

For policymakers

Optional addition:

...final invention as soon as it's invented (an AI system that can do anything a human can, including inventing the next generation of AI systems that are even smarter), then we had better make sure that...

When asked about the idea of an AI smarter than a human, people tend to say "not for another hundred years". And they've been saying the exact same thing for 50 years now. The same thing happened with airplanes, the discovery of the nervous system, computers, and nuclear bombs; and often within 5 years of the discovery. And the last three years of groundbreaking progress in AI has made clear that they've never been more wrong.

"If you have an untrustworthy general superintelligence generating [sentences] meant to [prove something], then I would not only expect the superintelligence to be [smart enough] to fool humans in the sense of arguing for things that were [actually lies]... I'd expect the superintelligence to be able to covertly hack the human [mind] in ways that I wouldn't understand, even after having been told what happened[, because a superintelligence is, by definition, at least as smart to humans as humans are to chimpanzees]. So you must have some belief about the s... (read more)

1Trevor12mo
Techxecutives
1Trevor12mo
don't read #1 or #4 and then use it as an indicator to ignore the rest, the other ones definitely have significant viability as one-liners. #4 isn't even that bad, if you really think about it. 2, 3, and 5-8 are all highly viable entries for one-liners. They are the perfect length.

GPT-3 told me the following: Super intelligent AI presents a very real danger to humanity. If left unchecked, AI could eventually surpass human intelligence, leading to disastrous consequences. We must be very careful in how we develop and control AI in order to avoid this outcome.

GPT-3 told me the following: Super intelligent AI presents a very real danger to humanity. If left unchecked, AI could eventually surpass human intelligence, leading to disastrous consequences. We must be very careful in how we develop and control AI in order to avoid this outcome

There is a certain strain of thinker who insists on being more naturalist than Nature. They will say with great certainty that since Thor does not exist, Mr. Tesla must not exist either, and that the stories of Asclepius disprove Pasteur. This is quite backwards: it is reasonable to argue that a machine will never think because the Mechanical Turk couldn't; it is madness to say it will never think because Frankenstein's monster could. As well demand that we must deny Queen Victoria lest we accept Queen Mab, or doubt Jack London lest we admit Jack Frost. Na... (read more)

You might think that AI risk is no big deal. But would you bet your life on it?

(policymakers)

Betting against the people who said pandemics were a big deal, six years ago, is a losing proposition.

(policymakers, tech executives)

(source)

Just because tech billionaires care about AI risk doesn't mean you shouldn't. Even if a fool says the sky is blue, it's still blue.

(policymakers, maybe ML researchers)

2NicholasKross2mo
The scare quotes around "tech billionaires" are unnecessary in this context, even when you are talking to someone who is totally anti-billionaire.
5Peter Berggren2mo
Fixed; thanks!

Hoping you'll run out of gas before you drive off a cliff is a losing strategy. Align AGI; don't count on long timelines.

(ML researchers)

(adapted from Human Compatible)

Intelligence has an inverse relationship with predictability. Cranking up an AI's intelligence might crank down its predictability. Perhaps it will plummet at an unpredictable moment.

For policymakers, but also the other two. It's hard to handle policymakers, because many policymakers might think that an AGI would fly to the center of the galaxy and duke it out with God/Yaweh, mano a mano.

If the media reported on other dangers like it reported on AI risk, it would talk about issues very differently. It would compare events in the Middle East to Tom Clancy novels. It would dismiss runaway climate change by saying it hasn't happened yet. It would think of the risk of nuclear war in terms of putting people out of work. It would compare asteroid impacts to mudslides. It would call meteorologists "nerds" for talking about hurricanes. 

AI risk is serious, and it isn't taken seriously. It's time to look past the sound bites and focus on what e... (read more)

Hey Siri, "Is there a God?" "There is now."

 - Adapted from Fredric Brown, "The Answer" - for policymakers.

1Peter Berggren2mo
This seems like it falls into the trap of being "too weird" for policymakers to take seriously. Good concept; maybe work on the execution a bit?
1Trevor12mo
Many policymakers might think that an AGI would fly to the center of the galaxy and duke it out with God/Yaweh, mano a mano. I've seen singularitarians who've had that dilemma. Religion doesn't really preclude any kind of intelligent thought or impede anyone from getting into any position, since it starts from birth, and rarely insists on any statement (other than powerful stipulations about what a superhuman entity is supposed to look like)

Humans are unpredictable. A hyperintelligent machine would notice this and see human control as fundamentally suboptimal, no matter the specifics of what's running under the hood.

AIs are like genies: they’ll fulfill the literal words of your wish, but don’t care at all about the spirit.

Once AI is close enough to human intelligence, it will be able to improve itself without human maintenance. It will be able to take itself the rest of the way, all the way up to humanlike intelligence, and it will probably pass that point as quickly as it arrived. There's no upper limit to intelligence, only an upper limit for intelligent humans; we don't know what a hyperintelligent machine would do, it's never happened before. If it had, we might not be alive right now; we simply don't know how something like that would behave, only that it would be as capable of outsmarting us as we are of outsmarting lions and hyenas.

Policymakers, and maybe techxecutives.

People said man wouldn't fly for a million years. Airplanes were fighting each other eleven years later. Superintelligent AI might happen faster than you think. (policymakers, tech executives) (source) (other source)

What if there was a second intelligent species on Earth, besides humans? The world's largest tech companies are working to build super-human AI, so we may find out sooner than you think. (Policymakers)

[open to critiques/rewordings so we don't accidentally ignore pretty-intelligent nonhuman animals, like dolphins.]

If we build an AI that's as smart as a human, that's not a human. There will be all sorts of things we did wrong, including deliberately, in order to optimize for performance. All someone has to do is crank it up 10x faster in order to get slightly better results, and that could be it. Something smarter than any human, thinking hundreds or millions of times faster than a human mind, able to learn or figure out anything it wants, including how to outmaneuver all human creators.

Techxecutives and ML researchers. I'd worry about sending this to policymakers, b... (read more)

We may be physically weaker than apes, but we're much smarter, so they don't really threaten us. Now, if a computer was much smarter than us... (Policymakers)

Superhuman, super-smart AI may seem like science fiction… but so did rockets and smartphones, once upon a time. (Policymakers, Tech executives)

If we can't keep advanced AI safe for us, it will keep itself safe from us... and we probably won't be happy with the result. (policymakers)

1Trevor12mo
...we can't keep a [limitlessly self-improving AI] safe for us, it will...
1Peter Berggren2mo
I thought that would ruin the parallelism and flow a bit, and this isn't intended for the "paragraph" category, so I didn't put that in yet.

One of my one-liners was written by an AI. You probably can't tell which one. Imagine what AI could do once it's even more powerful. (tech executives, ML researchers)

AI could be the best or worst thing to ever happen to humanity. It all depends on how we design and program them. (policymakers, tech executives)

Building advanced AI without knowing how to align it is like launching a rocket without knowing how gravity works. (tech executives, ML researchers)

(based on MIRI's "The Rocket Alignment Problem")

When AI’s smarter than we are, we have to be sure it shares our values: after all, we won’t understand how it thinks. (policymakers, tech executives)

Social media AI has already messed up politics, and AI's only getting more powerful from here on out. (policymakers)

In the same way, suppose that you take weak domains where the AGI can't fool you, and apply some gradient descent to get the AGI to stop outputting actions of a type that humans can detect and label as 'manipulative'.  And then you scale up that AGI to a superhuman domain.  I predict that deep algorithms within the AGI will go through consequentialist dances, and model humans, and output human-manipulating actions that can't be detected as manipulative by the humans, in a way that seems likely to bypass whatever earlier patch was imbued by gradie... (read more)

"Manipulating humans is a [winning] strategy if you've accurately modeled (even at quite low resolution) what humans are and what they do in the larger scheme of things."

EY

For Techxecutives, maybe ML researchers, depends on what they work on.

"I'm doubtful that you can have an AGI that's significantly above human intelligence in all respects, without it having the capability-if-it-wanted-to of looking over its own code and seeing lots of potential improvements."

EY

Techxecutives, maybe ML researchers

For the last 20 years, AI technology has improved randomly and without warning. We are now closer to human level artificial intelligence than ever before, to building something that can invent solutions to our problems, or invent a way to make itself as smart to humans as humans are to ants. But we won't know what it will look like until after it is invented; if it is as smart as a human but learns one thousand times as fast, it might detect the control system and compromise all human controllers. Choice and intent are irrelevant; it's a computer, all it t... (read more)

If an AI system becomes as smart to a human as a human is to a lion or cow, it will not have much trouble compromising its human controllers. After a certain level of intelligence, above human-level intelligence, AGI could thwart any manmade control system immediately after a single glitch, the same way that humans do things on a whim.

For policymakers, maybe techxecutives too.

An AI capable of programming might be able to reprogram itself smarter and smarter, soon surpassing humans as the dominant species on the planet.

2NicholasKross2mo
reworked version: AI systems are already [https://www.deepmind.com/blog/competitive-programming-with-alphacode] capable of programming [https://openai.com/blog/openai-codex/]. If humans can program smart AI, what could smart AI program? Smarter AI? Is this process controllable by humans?

I just won an election using nothing but AI-written speeches

"When we create an Artificial General Intelligence, we will be giving it the power to fundamentally transform human society, and the choices that we make now will affect how good or bad those transformations will be.  In the same way that humanity was transformed when chemist and physicists discovered how to make nuclear weapons, the ideas developed now around AI alignment will be directly relevant to shaping our future."

"Once we make an Artificial General Intelligence, it's going to try and achieve its goals however it can, including convincing everyone around it that it should achieve them.  If we don't make sure that it's goals are aligned with humanity's, we won't be able to stop it."

"The present situation can be seen as one in which a common resource, the remaining timeline until AGI shows up, is incentivized to be burned by AI researchers because they have to come up with neat publications and publish them (which burns the remaining timeline) in order to earn status and higher salaries"

Eliezer Yudkowsky

Bear in mind that in reality, there is a demand for cutting edge AI to be mounted on nuclear cruise missiles, and that no person, group, or movement can prevent the military from procuring the nuclear weapons technology that it claims ... (read more)

50 years ago was 1970. The gap between AI systems then and AI systems now seems pretty plausibly greater than the remaining gap [between current AI and AGI], 

Nate Soares

Optional extra:

even before accounting the recent dramatic increase in the rate of progress, and potential future increases in rate-of-progress as it starts to feel within-grasp.

"There are 7 billion people on this planet.  Each one of them has different life experiences, different desires, different aspirations, and different values.  The kinds of things that would cause two of us to act, could cause a third person to be compelled to do the opposite.  An Artificial General Intelligence will have no choice but to act from the goals we give it.  When we give it goals that 1/3 of the planet disagrees with, what will happen next?" (Policy maker)

All our ML systems optimize for some proxy of what we really want, like "high score" for playing a game well, or "ads clicked" for suggesting news sites. As AI gets better at optimizing the high score, the things imperfectly linked with the high score (like good gameplay or accurate journalism) get thrown aside. After all, it's not efficient to maximize a value that wasn't programmed in. (Machine learning researchers)

The big ML systems, like GPT-3 and MT-NLG, are taking increasingly large amounts of computing power to train. Normally, this requires expensive GPUs to do math simulating neural networks. But the human brain, better than any computer at a variety of tasks, is fairly small. Plus, it uses far less energy than huge GPU clusters!

As much as AI has progressed in the last few years, is it headed for an AI winter? Not with new hardware on the horizon. By mimicking the human brain, neuromorphic chips and new analog computers can do ML calculations faster than GPUs,... (read more)

"With all the advanced tools we have, and with all our access to scientific information, we still don't know the consequences of many of our actions.  We still can't predict weather more than a few days out, we can't predict what will happen when we say something to another person, we can't predict how to get our kids to do what we say. When we create a whole new type of intelligence, why should we be able to predict what happens then?"

"There are lots of ways to be smarter than people.  Having a better memory, being able to learn faster, having more thorough reasoning, deeper insight, more creativity, etc.  
When we make something smarter than ourselves, it won't have trouble doing what we ask.  But we will have to make sure we ask it to do the right thing, and I've yet to hear a goal everyone can agree on." (Policy maker)

Many in Silicon Valley (and China) think there's a huge advantage to inventing super-smart AI first, so they can reap the benefits and "pull ahead" of competitors. If an AI smarter than any human can think of military strategies, or hacking techniques, or terrorist plots, this is a terrifying scenario. (Policymakers)

What will happen if an AI realises that it is in a training loop? There are a lot of bad scenarios that could branch out from this point. Potentially this scenario sounds weird or crazy, however, humans possess the ability to introspect and philosophise on similar topics, even though our brains are “simply” computational apparatus which do not seem to possess any qualities that a sufficiently advanced AI could not possess.

"When we start trying to think about how to best make the world a better place, rarely can anyone agree on what is the right way.   Imagine what would happen to if the first person to make an Artificial General intelligence got to say what the right way is, and then get instructed how to implement it like they had every researcher and commentator alive spending all their day figuring out the best way to implement it.  Without thinking about if it was the right thing to do." (Policymaker)

"How well would your life, or the life of your child go, if you took your only purpose in life to be the first thing your child asked you to do for them?  When we make an Artificial Intelligence smarter than us, we will be giving it as well defined of a goal as we know how.  Unfortunately, the same as a 5 year old doesn't know what would happen if their parent dropped everything to do what they request, we won't be able to think through the consequences of what we request." (Policymaker)

3NicholasKross2mo
Better-worded version: "Back when you 5 years old, you may've asked your parents for a lot of things. What would've happened if they'd done everything you asked for? If humanity eventually invents smart AI, we'd have our own literal-minded 'parent' catering to our explicit demands. This could go very wrong."
1Robert Cousineau2mo
Nice work there!

"All scientific ignorance is hallowed by ancientness. Each and every absence of knowledge dates back to the dawn of human curiosity; and the hole lasts through the ages, seemingly eternal, right up until someone fills it."

EY, AI as a pos neg factor, 2006ish

Optional extra:

"I think it is possible for mere fallible humans to succeed on the challenge of building Friendly AI. But only if intelligence ceases to be a sacred mystery to us, as life was a sacred mystery to [scientists in 1922]. Intelligence must cease to be any kind of mystery whatever, sacred or not. We must execute the creation of Artificial Intelligence as the exact application of an exact art"

"Intelligence is not the first thing human science has ever encountered which proved difficult to understand. Stars were once mysteries, and chemistry, and biology. Generations of investigators tried and failed to understand those mysteries, and they acquired the reputation of being impossible to mere science"

EY, AI as a pos neg factor, 2006ish

Optional extra:

"Once upon a time, no one understood why some matter was inert and lifeless, while other matter pulsed with blood and vitality. No one knew how living matter reproduced itself, or why our hands obeyed our mental orders."

"The human mind is not the limit of the possible. Homo sapiens represents the first general intelligence. We were born into the uttermost beginning of things, the dawn of mind."

EY, AI as a pos neg factor, 2006ish

All three categories (most of my submissions go here, unless otherwise stated)

"It seems like an unfair challenge. Such competence is not historically typical of human institutions, no matter how hard they try. For decades the U.S. and the U.S.S.R. avoided nuclear war, but not perfectly; there were close calls, such as the Cuban Missile Crisis in 1962."

EY, AI as a pos neg factor, 2006ish

For policymakers (#1) and techxecutives (#2)

"Hominids have survived this long only because, for the last million years, there were no arsenals of hydrogen bombs, no spaceships to steer asteroids toward Earth, no biological weapons labs to produce superviruses, no recurring annual prospect of nuclear war... [and soon, we may have to worry about AGI as well, the finish line]. To survive any appreciable time, we need to drive down each risk to nearly zero.

EY, AI as a pos neg factor, 2006ish

For ML researchers only, since policymakers and techxecutives are often accustomed to "survival of the fittest" lifestyles

"Often Nature poses requirements that are grossly unfair, even on tests where the penalty for failure is death. How is a 10th-century medieval peasant supposed to invent a cure for tuberculosis? Nature does not match her challenges to your skill, or your resources, or how much free time you have to think about the problem."

EY, AI as a pos neg factor, 2006ish

Optional extra:

 And when you run into a lethal challenge too difficult for you, you die. It may be unpleasant to think about, but that has been the reality for humans, for thousands upon thousands ... (read more)

"If the pen is exactly vertical, it may remain upright; but if the pen tilts even a little from the vertical, gravity pulls it further in that direction, and the process accelerates. So too would smarter systems have an easier time making themselves smarter."

EY, AI as a pos neg factor

For policymakers, Techxecutives, and only ML researchers who are already partially sold on the "singularity" concept (or have the slightest interest in it).

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. 

I.J. Good, 1965 (he is dead and I don't have any claim to his share, even if he gets the entire share). FYI he was one of the first computer scientists and later on, he w... (read more)

If AI safety were a scam to raise money… it'd have way better marketing! (Adapted from here). (Machine learning researchers)

Every headline about an AI improvement is another warning of superhuman intelligence, and as a species, we're not ready for it. Crunch time is near. (Adapted from here). (Machine learning researchers)

Getting AI to understand human values, when we ourselves barely understand them, is a massive challenge for the world's best computer scientists, mathematicians, and philosophers. Are you one of them? (Adapted from here). (Machine learning researchers)

If AI can eventually be made smarter than humans, our fate would depend on it in the same way Earth's species depend on us. This is not reassuring for us. (Adapted from here). (Policymakers)

Humans are the smartest species on Earth… but we might still be the dumbest species that could've possibly built our technology. (Adapted from here). (Tech executives)

2Trevor12mo
...on earth... but the human mind might be the simplest possible mind that could have built technology as advanced as today's technology.

What are we supposed to do if we build something that's so vastly smarter than us that it can avoid being turned off? Certainly not turn it off.

What are we supposed to do with a worthless block that wants to be turned off, or randomly thinks up some reason to turn itself off? We try again.

"Who can say what science does not know? There is far too much science for any one human being to learn. Who can say that we are not ready for a scientific revolution, in advance of the surprise? And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI. Those two statements are not at all equivalent!"

EY, AI as a pos neg factor, 2006ish

1Trevor12mo
"And if wecannotmake progress on Friendly AI because we are not prepared, this does not mean we do notneedFriendly AI." Friendly AI will be the only thing we need, and unfriendly AI will be the worst possible thing that could happen. If the problem seems impossible to solve, that doesn't matter because this isn't about diminishing returns or ROI, it's about humanity's final chance at figuring things out. The payoff of a single working solution will be a payoff that continues yielding dividends for billions of years.

"The «ten-year rule» for genius, validated across fields ranging from math to music to competitive tennis, states that no one achieves outstanding performance in any field without at least ten years of effort. (Hayes 1981.) ... If we want people who can make progress on Friendly AI, then they have to start training themselves, full-time, years before they are urgently needed."

EY, AI as a pos neg factor, 2006ish

When will superhuman AI be invented? Experts disagree, but their estimates mostly range from a few years to a few decades away. The consensus is that artificial general intelligence, capable of beating humans at any mental task, will probably arrive before the end of the century. But, again, it could be just a few years away. With so much uncertainty, should't we start worrying about the new nonhuman intelligent beings now? (Adapted from here). (Machine learning researchers)

Right now, AI only optimizes for proxies of human values… is that good enough for the future? (Adapted from here). (Machine learning researchers)

2Trevor12mo
Please keep going! I had no idea that twitter would work for this, and even now that I know, I don't know how to use twitter so I can't even begin to find these sorts of things.

The failed promises [still] come swiftly to mind, both inside and outside the field of AI, when [AGI] is mentioned. The culture of AI research has adapted to this condition: There is a taboo against talking about human-level capabilities. There is a stronger taboo against anyone who appears to be claiming or predicting a capability they have not demonstrated with running code.

EY, AI as a pos neg factor, 2006ish

Policymakers and techxecutives might misunderstand or be put on their guard by these, so ML researchers only.

"If you deal out cards from a deck, one after another, you will eventually deal out the ace of clubs."

"We keep dealing through the deck until we deal out an [AGI], whether it is the ace of hearts or the ace of clubs."

EY, AI as a pos neg factor, 2006ish

Sharp, unanticipated jumps in intelligence are proven possible, known to occur frequently, and have a long history of taking place"

Derived from:

"I cannot perform a precise calculation using a precisely confirmed theory, but my current opinion is that sharp jumps in intelligence arepossible, likely, and constitute the dominant probability."

EY, AI as a pos neg factor, 2006ish

For policymakers and techxecutives:

"When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent [systems] -- on a still-shorter time scale."

Optional extra (for same non-ML audience)

"The best analogy I see is to the evolutionary past: Animals can adapt to problems and make inventions, but often no faster than natural selection can do its work -- the world acts as its own simulator in the case of natural selection. We h... (read more)

"history books tend to focus selectively on movements that have an impact, as opposed to the vast majority that never amount to anything. There is an element involved of luck, and of the public’s prior willingness to hear."

EY, AI as a pos neg factor, 2006ish

For ML researchers, as policymakers might have their own ideas about how history books end up written.

"To date [in 1993], there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes," then there is little doubt that [even] more intelligent [machines] can be constructed shortly thereafter."

Vinge, technological singularity, 1993

"I argue that confinement is intrinsically impractical. Imagine yourself locked in your home with only limited data access to the outside, to your masters. If those masters thought at a rate -- say -- one million times slower than you, there is little doubt that over a period of years (your time) you could come up with a way to escape. I call this "fast thinking" form of superintelligence "weak superhumanity." Such a "weakly superhuman" entity would probably burn out in a few weeks of outside time. "Strong superhumanity" would be more than cranking up the ... (read more)

1NicholasKross2mo
Nitpick: "singularity" is basically one abstract concept (exponential growth / infinity / asymptote) to another abstract concept (technology / economic growth / intelligence). So the intuition-pumping is probably a better first step (like examples of past technology that seemed absurd before it was invented).

"a wolf cannot understand how a gun works, or what sort of effort goes into making a gun, or the nature of that human power which lets us invent guns" 

Optional extra (for techxecutives and policymakers)

"Vinge (1993) wrote:

Strong superhumanity would be more than cranking up the clock speed on a human-equivalent mind. It’s hard to say precisely what strong superhumanity would be like, but the difference appears to be profound. Imagine running a dog mind at very high speed. Would a thousand years of doggy living add up to any human insight?"

EY, AI as a pos neg factor, 2006ish

Here is an example of a way that something could kill everyone if it suddenly became significantly smarter than a human:

  • Crack the protein folding problem, to the extent of being able to generate DNA strings whose folded peptide sequences fill specific functional roles in a complex chemical interaction.
  • Email sets of DNA strings to one or more online laboratories which offer DNA synthesis, peptide sequencing, and FedEx delivery. (Many labs currently offer this service, and some boast of 72-hour turnaround times.)
  • Find at least one human connected to the Inter
... (read more)

"The future has a reputation for accomplishing feats which the past thought impossible"

Optional extra:

"Future civilizations have even broken what past civilizations thought (incorrectly, of course) to be the laws of physics. If prophets of 1900 AD — never mind 1000 AD — had tried to bound the powers of human civilization a billion years later, some of those impossibilities would have been accomplished before the century was out; transmuting lead into gold, for example"

EY, AI as a pos neg factor, 2006ish