Haiku — LessWrong

Minimum required pacing: finish aligning it before you train the next one

Sufficient alignment of AI systems at one level of capability does not necessarily provide significant evidence that they will be sufficiently aligned at the next level of capability. This is bad enough because alignment is not the first step in (or robustly integrated throughout) the training process, and a misaligned AI might escape during training. But it's especially bad because of threshold effects. We don't understand reflexive stability. We don't understand ontological shift. We don't know what all can go wrong (or how or why) once an AI system know... (read more)

Jimrandomh's Shortform

Haiku8d236

Dario consistently centering China as the boogeyman is perhaps the most damaging thing any of the AI companies have ever done.

Shortform

Haiku10d6-3

Stop doing "safety" work for the uncontrollable doomsday weapon companies. That is part of what gets everyone killed.

leogao's Shortform

[+]Haiku14d-40-51

leogao's Shortform

Haiku14d10

Setting aside the parts of math that have utility, that stance is fundamentally anti-human. Mathematics is full of intrinsically useless and arbitrary truths. Uncovering those truths is a human endeavor, as are all endeavors toward intrinsically useless and arbitrary things. Loving and pursuing such things is an important part of what it is to be human. The whole point is the experience of meaning. A machine that can spit out any intrinsically useless and arbitrary truth on command would destroy a source of meaning for many people, while in return providin... (read more)

Rauno's Shortform

Haiku21d-3-6

They would deserve credit for completely shutting down all R&D on the "kill my family" machine. I'm ready and waiting to praise them if they ever did just that one thing.

Alexa Pan's Shortform

Haiku25d10

I continue to insist that we have a threshold problem, not just a speed problem. We currently have no way to constrain what will happen when AI becomes strictly superhuman at creating better AI, or when it gains the option to permanently escape human oversight or disempower humanity. We have no guarantees of the degree of generalization from our intent at those thresholds. They are opaque walls to us.

The continuity argument fails at all of these steps, not just new architectures. But if AI does an increasing amount of the AI research, we'll also have no wa... (read more)

If Anyone Builds It, Everyone Dies: One Year Closer

Haiku26d54

I vibe with the sentiment, but I'm going to be annoying and disagree with that specific framing. There is always a non-zero probability of human extinction from all kinds of things, including nuclear war and bioweapons, as well as asteroid impacts or a sufficiently bad supervolcano. (Zero isn't a real probability in Bayesian epistemology, but that's another rabbit hole.)

That said, it's perfectly sane to not accept even very low probabilities of extinction from AI. Some people draw the line at 1 in 20, others at 1 in a billion. I don't know exactly where I ... (read more)

If Anyone Builds It, Everyone Dies: One Year Closer

Haiku26d42

To the degree that we can provide quantitative guarantees of next-generation AI behavior, it should be fine to take things up another notch, modulo all the other ethical questions involved. Right now we don't have a way of doing that. We can't even properly measure the safety of existing systems, let alone a hypothetical next one.

ASI does necessarily escape every meaningful constraint, at least until we solve a series of extremely hard scientific problems that we are nowhere near solving. There are a few ways to notice the difficulty here. I'll take a crac... (read more)

If Anyone Builds It, Everyone Dies: One Year Closer

Haiku1mo65

Researchers and philosophers have had that exact debate for something like 20 years, and I'm not encouraged by the details of the discussion.

Taking a step back, getting a planet full of innocent people into that situation to begin with is indefensible, by the lights of the vast majority of those people.

If Anyone Builds It, Everyone Dies: One Year Closer

Haiku1mo137

Ruling out bad behavior is necessary if you're creating something that you will have no way of controlling or constraining, which is what ASI implies. There's "advanced AI" and then there's "the new most capable species on the planet, under which you live at its whim." No one has any idea how to prevent the former from becoming the latter, other than shutting down all frontier AI R&D on the planet for at least a good while, which is the only sane and morally defensible route we can take.

The AI Risk Network

Haiku1mo21

The naming confused me so much. Liron Shapira of Doom Debates (with the Doom Train) does a show called Warning Shots in partnership with a YouTube channel called the AI Risk Network, which has several shows.

Doom as a bad method not a utopia trade-off

Haiku1mo30

That makes sense; it was easy to read your comment as applying to individual actors, so I appreciate the clarification. (And for extra clarity on my part, I wasn't meaning to making an accusation against your character even in my initial reading of your statement, which is analytical and neutral. My comment was very pointed, but was not intended to be pointed at you.)

Doom as a bad method not a utopia trade-off

Haiku1mo87

That makes sense for personal risk, but when it comes to other people, it's not your call. Unilaterally and knowingly exposing the entire world to extreme amounts of additional risk is deeply immoral, of the "doesn't believe in consent" variety.

testingthewaters's Shortform

Haiku1mo76

It doesn't have to be up to them, and by no rights should it be. If the US government understood what was happening, OpenAI et al would be shut down overnight.

I'm just some guy, and one of many doing grassroots lobbying about AI extinction risk. And yet, if there had been just 200 more people having my very modest level of impact on US lawmakers over the past 2 years, I think we would have a global AI treaty today, or be very close.

Other countries have less impact, but far from zero! The impact of multiple middle powers formally recognizing that superintelligent AI is an extinction level threat, may be very close, and should be banned would be very substantial.

Ian McKenzie's Shortform

Haiku1mo10

I think you can get people unstuck from that prior if you offer them a line of retreat toward an even bigger reason to hate the AI companies.

My current go-to snappy retort for use in low-quality discourse (e.g. YouTube comments) is: "Exactly. Just like how oil companies invented climate change to make oil sound cool to investors." (I don't have any evidence that this works, but I predict that little nudges and intuition pumps like this would work over time if enough people offer them.)

Ian McKenzie's Shortform

Haiku1mo42

But then why doesn't Cal get into PauseAI?

Warning: Incautious psychoanalysis ahead

PauseAI is trying to solve a problem, and people like Cal aren't in the business of trying to solve problems.

It's useful to notice here that Cal Newport and Ed Zitron are very similar figures. Cal has expertise in non-AI computer science and can sometimes make a coherent argument, so he's the shiny variant. But realistically, neither of them have a clue about AI, and they both give off an aura of religious zeal that prevents things like "evidence" from having any effect on... (read more)

Let's talk about the AI coordination problem

Haiku1mo32

The issue is a willful lack of curiosity about verification mechanisms, which are viable in the very near term.

Daniel Kokotajlo's Shortform

Haiku1mo1-2

I can imagine a scientific reason for doing so: learning more about how cyber capabilities scale, in order to get empirical data on the details of their emergence, in order to better design training runs that don't result in dangerous cyber capabilities.

(Though I really doubt this is what they were doing.)

Thomas Kwa's Shortform

Haiku2mo1415

Why not just throw all of your effort into making RSI illegal?