Sorted by New

Wiki Contributions


The title is definitely true: humans are irrational in many ways, and an AGI will definitely know that. But I think the conclusions from this are largely off-base.

  • the framing of "it must conclude we are inferior" feels like an unnecessarily human way of looking at it? There are a bunch of factual judgements anyone analyzing humans can make, some that have to do with ways we're irrational and some that don't (can we be trusted to keep to bargains if one bothered to make them? maybe not. Are we made of atoms which could be used for other things? yes); once you've established these kinds of individual questions, I'm not sure what you gain by summing it up with the term 'inferior', or why you think that the rationality bit in particular is the bit worth focusing on when doing that summation.

  • there is nothing inherently contradictory in a "perfectly rational" agent serving the goals of us "irrational" agents. When I decide that my irrational schemes require me to do some arithmetic, the calculator I pick up does not refuse to function on pure math just because it's being used for something "irrational". (Now whether we can get a powerful AGI to do stuff for us is another matter; "possible" doesn't mean we'll actually achieve it.)

Typo: "fit the bill" and "fitting the bill" -> "foot the bill" and "footing the bill"

We humans dominate the globe but we don’t disassemble literally everything (although we do a lot) to use the atoms for other purposes.

There are two reasons why we don't:

  1. We don't have the resources or technology to. For example there are tons of metals in the ground and up in space that we'd love to get our hands on but don't yet have the tech or the time to do so, and there are viruses we'd love to destroy but we don't know how. The AGI is presumably much more capable than us, and it hardly even needs to be more capable than us to destroy us (the tech and resources for that already exist), so this reason will not stop it.

  2. We don't want to. For example there are some forests we could turn into useful wood and farmland, and yet we protect them for reasons such as "beauty", "caring for the environment", etc. Thing is, these are all very human-specific reasons, and:

Isn’t it arguable that ASI or even AGI will have a better appreciation for systems ecology than we do…

No. Sure it is possible, as in it doesn't have literally zero chance if you draw a mind at random. (Similarly a rocket launched in a random direction could potentially land on the moon, or at least crash into it.) But there are so many possible things an AGI could be optimizing for, and there is no reason that human-centric things like "systems ecology" should be likely, as opposed to "number of paperclips", "number of alternating 1s and 0s in its memory banks", or an enormous host of things we can't even comprehend because we haven't discovered the relevant physics yet.

(My personal hope for humanity lies in the first bullet point above being wrong: given surprising innovations in the past, it seems plausible that someone will solve alignment before it's too late, and also given some semi-successful global coordination things in the past (avoiding nuclear war, banning CFCs), it seems plausible that a few scary pre-critical AIs might successfully galvanize the world into successful delaying action for long enough that alignment could be solved)

(This is of course just my understanding of his model, but) yes. The analogy he uses is that while you cannot predict Stockfish's next move in chess, you can predict for 'certain' that it will win the game. I think the components of the model are roughly:

  • it is 'certain' that, given the fierce competition and the number of players and the incentives involved, somebody will build an AGI before we've solved alignment.
  • it is 'certain' that if one builds an AGI without solving alignment first, one gets basically a random draw from mindspace.
  • it is 'certain' that a random draw from mindspace doesn't care about humans
  • it is 'certain' that, like Stockfish, this random draw AGI will 'win the game', and that since it doesn't care about humans, a won gameboard does not have any humans on it (because those humans were made of atoms which could be used for whatever it does care about)

part of this will be to make at least a nominal trade in every market I would want to be updated on

I don’t want to get updates on this so I’m sitting it out

For the future, you can actually follow/unfollow markets on Manifold independently of whether you've bet on them: they've moved this button around but it's currently on the "..." ("Market details") popup (which is found in the upper-right corner of a market's main pane).

I bought M270 of NO, taking out a resting buy order at 71%, causing me to note that you can sometimes get a better price buying in multiple steps where that shouldn’t be true. Weird.

According to the trade log at, it looks like you bought 230 of NO (75%->70%), then a bot user responded in under a second by buying YES back up to 72%, and then you bought your final 40 of NO at the new price. There are several bots which appear to be watching and counteracting large swings, so I sometimes purposefully purchase in two stages like this to take advantage of exactly this effect. (Though I don't know the bots' overall strategies, and some appear to sometimes place immediate bets in the same direction as a large swing instead.)

The range of possible compute is almost infinite (e.g. 10^100 FLOPS and beyond). Yet both intelligences are in the same relatively narrow range of 10^15 - 10^30

10^15 - 10^30 is not at all a narrow range! So depending on what the 'real' answer is, there could be as little as zero discrepancy between the ratios implied by these two posts, or a huge amount. If we decide that GPT-3 uses 10^15 FLOPS (the inference amount) and meanwhile the first "decent" simulation of the human brain is the "Spiking neural network" (10^18 FLOPS according to the table), then the human-to-GPT ratio is 10^18 / 10^15 which is almost exactly 140k / 175. Whereas if you actually need the single molecules version of the brain (10^43 FLOPS), there's suddenly an extra factor of ten septillion lying around.

So if you strongly opine that there should be a lot more legal immigration and a lot more enforcement, that's "no strong opinion"? This is not satisfying, though of course there'll never be a way to make the options satisfy everyone <shrug>.

You might also be able to do even the original setup for non-trivially cheaper (albeit still expensive) by just offering a lower wage? Given that for most of the time they can be doing whatever they want on their own computer (except porn, evidently; what's so bad about that anyway :P ) and that you were able to find 5 people at the current rate, I'd guess you could fill the position with someone(s) you're happy with at 10% or even 20% less.

What does one answer for "How would you describe your opinion on immigration?" if your preferred policy would include much more legal immigration and more enforcement against illegal immigration?

we may adopt never interrupt someone when they're doing something you want, even for the wrong reason.

Don't take this too far though; calling out bullshit may still be valuable even if it's bullshit that temporarily supports "your side": ideally doing so helps raise the sanity waterline overall, and more cynically, bullshit may support your opponents tomorrow and you'll be more credible calling it bullshit then if you also do so today.

But in this exact case, banning GoF seems good and also not using taxpayer dollars on negative-EV things seems good, so these Republicans are just correct this time.

Load More