Reality, unlike fiction, doesn't need to have verisimilitude. They are persuaded already and racing towards the takeover.

What's the last model you did check with, o1-pro?

For alphazero, I want to point out that it was announced 6 years ago (infinity by AI scale), and from my understanding we still don't have a 1000x faster version, despite much interest in one.

I don't know the details, but whatever the NN thing (derived from Lc0, a clone of AlphaZero) inside current Stockfish is can play on a laptop GPU.

And even if AlphaZero derivatives didn't gain 3OOMs by themselves it doesn't update me much that that's something particularly hard. Google itself has no interest at improving it further and just moved on to MuZero, to AlphaFold etc.

The first one, why?

Do you have a more concrete example? Preferably the one from the actual EA causes.

How one should signal their decision procedure in real life without getting their ass busted for "gambling with lives" etc.?

Getting stuff formally specified is insanely difficult, thus unpractical, thus pervasive verified software is impossible without some superhuman help. Here we go again.

Even going from "one simple spec" to "two simple spec" is a huge complexity jump: https://www.hillelwayne.com/post/spec-composition/

And real-world software has a huge state envelope.

Even if that's the case, the amount of 0-days out there (and just generally shitty infosec landscape) is enough to pwn almost any valuable target.

While I'd appreciate some help to screen out the spammers and griefers, this doesn't make me feel safe existentially.

Eliezer believes humans aligning superintelligent AI to serve human needs is as unsolvable as perpetual motion.

I'm confused. He said many times that alignment is merely hard*, not impossible.

• Especially with current constraints from hell.

I'm getting the same conclusions.

Think of a company like Google: building the biggest and best model is immensely valuable in a global, winner-takes-all market like search.

And this is in a world, where Google already announced that they're going to build even bigger model of their own

We are not, and won't for some* time.

• We have to upgrade our cluster with a fresh batch of Nvidia gadgets.

I doubt that any language less represented than English (or JS/Python) would be better since the amount of good data to ingest would be much less for them.