steven0461

LESSWRONG
LW

steven0461 — LessWrong

If we get things right, AI could have huge benefits

8mo

There is no contradiction between AI carrying huge potential risks, and it carrying huge potential upsides if we navigate the risks. Both are a consequence of the prospect of AI becoming extremely powerful.

The benefits that human-aligned AGI could bring are a major part of what motivates researchers to build such systems. Things we can confidently expect based on extrapolating from current technology include:

The development of more advanced technology in fields like agriculture, medicine, energy, computing, and transportation.
Economic growth in the form of widespread access to services that are currently provided by remote workers.
Robots doing much of our manual labor.

Examples that are more speculative, but seem physically feasible, include:

Ending diseases.
Curing aging and involuntary

Algon

Algon, steven0461, Vishakha

8mo

Context: This is a linkpost for https://aisafety.info/questions/NM3G/10:-Advanced-AI-is-a-big-deal-even-if-we-don%E2%80%99t-lose-control

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

So far, we’ve discussed one class of possible consequences of advanced AI: systems ending up with different values than we intended, and autonomously deciding to seize control. This is the main concern that this site is about.

But of course, introducing a highly powerful class of tools into the world, let alone a new intelligent species, will have many other kinds of implications:

Even today’s systems are already quite useful, and at the same time cause significant harms. For example,

... (read 335 more words →)

Defeat may be irreversibly catastrophic

Vishakha

Vishakha, Algon, steven0461

8mo

Context: This is a linkpost for https://aisafety.info/questions/NM3P/9:-Defeat-may-be-irreversibly-catastrophic

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

When you imagine a global catastrophe, maybe the kind of event that comes to mind is one that strikes a big blow against human civilization but doesn’t ruin it permanently. Events like pandemics, environmental disasters, and even nuclear wars would cause enormous suffering and millions or billions of deaths, but would probably leave part of human civilization alive to recover. Future generations could still flourish.

An AI takeover catastrophe is not like that. An inherent feature of AI taking... (read 342 more words →)

AI can win a conflict against us

Algon

Algon, steven0461, Vishakha

8mo

Context: This is a linkpost for https://aisafety.info/questions/NM3O/8:-AI-can-win-a-conflict-against-us

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Suppose an AI has realized that controlling the world would let it achieve its aims. If it’s weak, it might just obey humans. If it’s roughly at our power level, it might make deals or trades with us. But if it’s sufficiently capable, it might make a play to seize control from us. That is, it might attempt takeover. Could it succeed?

Intelligence confers a kind of power. (Keep in mind that by intelligence, we don’t just mean clever puzzle-solving,... (read 430 more words →)

Different goals may bring AI into conflict with us

Algon

Algon, steven0461, Vishakha

8mo

Context: This is a linkpost for https://aisafety.info/questions/NM3H/7:-Different-goals-may-bring-AI-into-conflict-with-us

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Aligning the goals of AI systems with our intentions could be really hard. So suppose we fail, and build a powerful AI that pursues goals different from ours. You might think it would just go off and do its own thing, and we could try again with a new AI.

But unfortunately, according to the idea of instrumental convergence, almost any powerful system that optimizes the world will find it useful to pursue certain kinds of strategies. And these strategies... (read 348 more words →)

AI’s goals may not match ours

Algon

Algon, steven0461, Vishakha

9mo

Context: This is a linkpost for https://aisafety.info/questions/NM3I/6:-AI%E2%80%99s-goals-may-not-match-ours

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Making AI goals match our intentions is called the alignment problem.

There’s some ambiguity in the term “alignment”. For example, when people talk about “AI alignment” in the context of present-day AI systems, they generally mean controlling observable behaviors like: Can we make it impossible for the AI to say ethnic slurs? Or to advise you how to secretly dispose of a corpse? Although such restrictions are sometimes circumvented with "jailbreaks", on the whole, companies mostly do manage to... (read 625 more words →)

AI may pursue goals

Algon

Algon, steven0461, Vishakha

9mo

Context: This is a linkpost for https://aisafety.info/questions/NM3J/5:-AI-may-pursue-goals

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Suppose that, as argued previously, in the next few decades we’ll have superintelligent systems. What role will they play?

One way to imagine these systems is purely as powerful and versatile tools, similar to most current systems. They could take broad directions from humans about what actions to take or what questions to answer, and cleverly fill in the details.

But another way is as agents, operating autonomously in the world. They could have their own goals — some kinds of futures... (read 269 more words →)

The road from human-level to superintelligent AI may be short

Vishakha

Vishakha, Algon, steven0461

10mo

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

It’s tempting to think that, even if AI can become vastly smarter than humans, getting to that point will take a vast amount of time, like centuries. However:

As discussed before, inputs to AI progress — including investment, hardware quality, and software efficiency — are growing exponentially. Exponential growth can be deceptively fast. AI that fully substitutes for humans would bring returns on the scale of the whole economy, as well

... (read 369 more words →)

Human-level is not the limit

Vishakha

Vishakha, Algon, steven0461

10mo

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

We’ve built technologies like skyscrapers that are much larger than us, bulldozers that are much stronger, and airplanes that are much faster. Similarly, there are good reasons to think that machines can eventually become much more capable at general cognitive problem-solving.

On an abstract level, our intelligence came from evolution, and while evolution results in well-optimized systems, it can’t “think ahead” or deliberately plan design choices. Humans are just the first... (read 543 more words →)

AI may attain human-level soon

Vishakha

Vishakha, Algon, steven0461

10mo

This is an article in the new intro to AI safety series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

The main companies building AI, such as OpenAI, Anthropic, and Google DeepMind, are explicitly aiming for artificial general intelligence: AI that has the same kind of general reasoning ability that humans have, and can do all the same tasks and jobs (at least the ones that don’t require a physical body).

We’re not quite there yet, and predicting technological progress is hard. But there are a few reasons to think we'll... (read 461 more words →)

Replying toContra papers claiming superhuman AI forecasting

steven04611y

Contra papers claiming superhuman AI forecasting

As I understand it, the Metaculus crowd forecast performs as well as it does (relative to individual predictors) in part because it gives greater weight to more recent predictions. If "superhuman" just means "superhumanly up-to-date on the news", it's less impressive for an AI to reach that level if it's also up-to-date on the news when its predictions are collected. (But to be confident that this point applies, I'd have to know the details of the research better.)

Replying toThis is already your second chance

steven04612y

This is already your second chance

"Broekveg" should be "Broekweg"

Replying toSuperforecasting the premises in “Is power-seeking AI an existential risk?”

steven04612y

Superforecasting the premises in “Is power-seeking AI an existential risk?”

partly as a result of other projects like the Existential Risk Persuasion Tournament (conducted by the Forecasting Research Institute), I now think of it as a data-point that “superforecasters as a whole generally come to lower numbers than I do on AI risk, even after engaging in some depth with the arguments.”

I participated in the Existential Risk Persuasion Tournament and I disagree that most superforecasters in that tournament engaged in any depth with the arguments. I also disagree with the phrase "even after arguing about it" - barely any arguing happened, at least in my subgroup. I think much less effort went into these estimates than it would be natural to assume based on how the tournament has been written about by EAs, journalists, and so on.

Replying toStampy's AI Safety Info soft launch

steven04612y*

Stampy's AI Safety Info soft launch

Thanks, yes, this is a helpful type of feedback. We'll think about how to make that section make more sense without background knowledge. The site is aimed at all audiences, and this means we'll have to navigate tradeoffs about text leaving gaps in justifying claims vs. being too long vs. not having enough scope to be an overview. In this case, it does look like we could make the tradeoff on the side of adding a bit more text and links. Your point about the glossary sounds reasonable and I'll pass it along. (I guess the tradeoff there is people might see an unexplained term and not realize that an earlier instance of it had a glossary link.)

Replying toStampy's AI Safety Info soft launch

steven04612y

Stampy's AI Safety Info soft launch

You're right that it's confusing, and we've been planning to change how collapsing and expanding works. I don't think specifics have been decided on yet; I'll pass your ideas along.

I don't think there should be "random" tabs, unless you mean the ones that appear from the "show more questions" option at the bottom. In some cases, the content of child questions may not relate in an obvious way to the content of their parent question. Is that what you mean? If questions are appearing despite not 1) being linked anywhere below "Related" in the doc corresponding to the question that was expanded, or 2) being left over from a different question that was expanded earlier, then I think that's a bug, and I'd be interested in an example.

Replying toStampy's AI Safety Info soft launch

steven04612y

Stampy's AI Safety Info soft launch

Quoting from our Manifund application:

We have received around $46k from SHfHS and $54k from LTFF, both for running content writing fellowships. We have been offered a $75k speculation grant from Lightspeed Grants for an additional fellowship, and made a larger application to them for the dev team which has not been accepted. We have also recently made an application to Open Philanthropy.

Replying toStampy's AI Safety Info soft launch

steven04612y

Stampy's AI Safety Info soft launch

EA Forum version (manually crossposting to make coauthorship work on both posts):

https://forum.effectivealtruism.org/posts/mHNoaNvpEuzzBEEfg/stampy-s-ai-safety-info-soft-launch

Replying toJoin AISafety.info's Writing & Editing Hackathon (Aug 25-28) (Prizes to be won!)

steven04612y

Join AISafety.info's Writing & Editing Hackathon (Aug 25-28) (Prizes to be won!)

if there's interest in finding a place for a few people to cowork on this in Berkeley, please let me know

Replying toStampy's AI Safety Info - New Distillations #4 [July 2023]

steven04612y

Stampy's AI Safety Info - New Distillations #4 [July 2023]

Thanks, I made a note on the doc for that entry and we'll update it.

Replying toStampy's AI Safety Info - New Distillations #4 [July 2023]

steven04612y

Stampy's AI Safety Info - New Distillations #4 [July 2023]

Traffic is pretty low currently, but we've been improving the site during the distillation fellowships and we're hoping to make more of a real launch soon. And yes, people are working on a Stampy chatbot. (The current early prototype isn't finetuned on Stampy's Q&A but searches the alignment literature and passes things to a GPT context window.)

Are there online spaces that talk about the same stuff LW talks about (AI futurism, technical rationality, and so on), with reasonably high quality standards, but more conversational-oriented and less soapbox-oriented, and maybe with less respectability signaling? I often find myself wanting to talk about things discussed here but feeling overconstrained by things like knowing that comments are permanent and having to anticipate objections instead of taking them as they come.

What's the name of the proto-fallacy that goes like "you should exchange your oranges for pears because then you'll have more pears", suggesting that the question can be resolved, or has already been resolved, without ever considering the relative value of oranges and pears? I feel like I see it everywhere a lot, including on LW.

According to electionbettingodds.com, this morning, Trump president 2024 contracts went up from about 0.18 to 0.31 on FTX but not elsewhere. Not sure what's going on there or if people can make money on it.

There's a meme in EA that climate change is particularly bad because of a nontrivial probability that sensitivity to doubled CO2 is in the extreme upper tail. As far as I can tell, that's mostly not real. This paper seems like a very thorough Bayesian assessment that gives 4.7 K as a 95% upper bound, with values for temperature rise by 2089 quite tightly constrained (Fig 23). I'd guess this is an overestimate based on conservative choices represented by Figs 11, 14, and 18. The 5.7 K 95% upper bound after robustness tests comes from changing the joint prior over feedbacks to create a uniform prior on sensitivity, which as far as... (read more)

Thinking out loud about some arguments about AI takeoff continuity:

If a discontinuous takeoff is more likely to be local to a particular agent or closely related set of agents with particular goals, and a continuous takeoff is more likely to be global, that seems like it incentivizes the first agent capable of creating a takeoff to make sure that that takeoff is discontinuous, so that it can reap the benefits of the takeoff being local to that agent. This seems like an argument for expecting a discontinuous takeoff and an important difference with other allegedly analogous technologies.

I have some trouble understanding the "before there are strongly self-improving AIs there will be moderately self-improving AIs" argument for continuity. Is there any reason to think the moderate self-improvement ability won't be exactly what leads to the strong self-improvement ability? Before there's an avalanche, there's probably a smaller avalanche, but maybe the small avalanche is simply identical to the early part of the large avalanche.

Where have these points been discussed in depth?

Are We Approaching an Economic Singularity? Information Technology and the Future of Economic Growth (William D. Nordhaus)

Has anyone looked at this? Nordhaus claims current trends suggest the singularity is not near, though I wouldn't expect current trends outside AI to be very informative. He does seem to acknowledge x-risk in section Xf, which I don't think I've seen from other top economists.

I'd like to register skepticism of the idea of a "long reflection". I'd guess any intelligence that knew how to stabilize the world with respect to processes that affect humanity's reflection about its values in undesirable ways (e.g. existential disasters), without also stabilizing it with respect to processes that affect it in desirable ways, would already understand the value extrapolation problem well enough to take a lot of shortcuts in calculating the final answer compared to doing the experiment in real life. (You might call such a calculation a "Hard Reflection".)

Has anyone tried to make complex arguments in hypertext form using a tool like Twine? It seems like a way to avoid the usual mess of footnotes and disclaimers.

There's been some discussion of tradeoffs between a group's ability to think together and its safety from reputational attacks. Both of these seem pretty essential to me, so I wish we'd move in the direction of a third option: recognizing public discourse on fraught topics as unavoidably farcical as well as often useless, moving away from the social norm of acting as if a consideration exists if and only if there's a legible Post about it, building common knowledge of rationality and strategic caution among small groups, and in general becoming skilled at being esoteric without being dishonest or going crazy in ways that would have been kept in check by larger audiences. I think people underrate this approach because they understandably want to be thought gladiators flying truth as a flag. I'm more confident of the claim that we should frequently acknowledge the limits of public discourse than the other claims here.

How much should I worry about the unilateralist's curse when making arguments that it seems like some people should have already thought of and that they might have avoided making because they anticipated side effects that I don't understand?

LESSWRONG
LW

LESSWRONG
LW

Eliezer Yudkowsky Facts

Stampy's AI Safety Info soft launch

Bayesian Adjustment Does Not Defeat Existential Risk Charity

AISafety.info "How can I help?" FAQ

steven0461

steven0461

If we get things right, AI could have huge benefits

Advanced AI is a big deal even if we don’t lose control

Defeat may be irreversibly catastrophic

AI can win a conflict against us

Different goals may bring AI into conflict with us

AI’s goals may not match ours

AI may pursue goals

steven0461

Eliezer Yudkowsky Facts

Stampy's AI Safety Info soft launch

Bayesian Adjustment Does Not Defeat Existential Risk Charity

AISafety.info "How can I help?" FAQ

steven0461

steven0461

If we get things right, AI could have huge benefits

Advanced AI is a big deal even if we don’t lose control

Defeat may be irreversibly catastrophic

AI can win a conflict against us

Different goals may bring AI into conflict with us

AI’s goals may not match ours

AI may pursue goals