Scamming is adversarial, so it's normal for a scam to appear like it's safe. But I'm not claiming my pattern match is superior to yours.
A scam could include getting financial information to get money, or a voice print for impersonation. Maybe the scammer has insider information about UFOs. Maybe it's entrapment for breaking gambling laws. Maybe a journalist is writing a story about how evil rationalists exploit innocent people with fringe beliefs for money.
The scam probability doesn't have to be large for it to dwarf the apparent benefits.
This pattern matches to anonymous person on the internet offering free money, which is typically a scam. Safer to pass, I think.
I'm curious about your derailment odds. The definition of "transformative AGI" in the paper is restrictive:
AI that can quickly and affordably be trained to perform nearly all economically and strategically valuable tasks at roughly human cost or less.
A narrow superintelligence that can, for example, engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition. I guess that would qualify as "AI-caused delay"? To follow the paper's model, we need to estimate these odds in a conditional world where humans are not regulating AI use in ways that significantly delay the path to transformative AGI, which further increases the risk.
Here is the world I am most interested in, where the conditional probability seems least plausible:
In this world, what is the probability that we were "derailed" by wars, such as China invading Taiwan?
Reading the paper naively, it says that there is a 30% chance that we achieved all of this technical progress, in the 99th percentile of possible outcomes, despite China invading Taiwan. That doesn't seem like a 30% chance to me. Additionally, if China invaded Taiwan, but it didn't prevent us achieving all this technical progress, in what sense was it a derailment? The executive summary suggests:
Even if an invasion doesn’t spark war, the sanctions applied in retaliation will shut down most of the world’s AI chip production
No, it can't possibly derail AI by shutting down chip production, in this conditional branch, because we already know from item 5 that we massively scaled chip production, and both things can't be true at the same time.
I get the same numbers on the web app, but I don't see how it relates to my comment, can you elaborate?
If there are no technical barriers, they are estimating a 37.7% chance of transformative AGI (which they estimate is a 5 to 50% extinction risk once created) and a 62.3% chance of "derailment". Some of the "derailments" are also extinction risks.
if there is no risk of derailment, the technical barriers make the probability 1.1%.
I don't think we can use the paper's probabilities this way, because technical barriers are not independent of derailments. For example, if there is no risk of severe war, then we should forecast higher production of chips and power. This means the 1.1% figure should increase.
Interesting. Using the snake-creating deity setting, what should I expect as a newly created sightless snake, waiting for my eyes? Suppose that the deity will answer my questions while I wait for the dice roll.
SIA: I expect that the deity has created an indescribably large number of batches, and has not rolled snake eyes yet. I expect that there is a 1/36 chance that they will roll snake eyes this time. If they don't, they will likely roll up more batches, and those probabilities will be pretty normal. And then I'll end up on a indescribably large world with an indescribable number of snakes, of which 50% are red-eyed.
SSA: I expect that the deity has created several batches of snakes, within the expected bounds of a Poisson distribution with P=1/36. I expect that there is a 50% chance that they will roll snake eyes this time, because while the dice are fair, if the deity rolls snake eyes then I am about 35x more real. And then I'll end up on a large world with lots of snakes (eg, 2^36), of which just over 50% are red-eyed.
So under SIA the past is shaped by anthropics, and under SSA the future is shaped by anthropics. And whatever happens I get very compelling evidence on the SSA vs SIA question.
This is exactly right under SIA, thanks. Under SIA, almost all the snakes exist in the final scenario, and therefore the limit is 50% as n -> infinity.
Under SSA it's a bit higher than 50%, because we always have a 1/36 chance of there being a single red-eyed snake.
The SSA vs SIA debate impacts both questions, but once you pick one of those then in Sleeping Beauty there's a clear answer of 1/2 or 1/3, whereas in this problem the infinities continue to make it unclear what the probability should be.
I definitely agree on the need for care in switching between variants. It can also be helpful that they can "change the situation" because this can reveal something unspecified about the original variant. Certainly I was helped by making a second variant, as this clarified for me that the probabilities are different from the deity view vs the snake view, because of anthropics.
In the original variant, it's not specified when exactly players get devoured. Maybe it is instant. Maybe everyone is given a big box that contains either a bazillion dollars, or human-eating snakes, and it opens exactly a year later.
In my variant, I was imagining the god initially created a batch of snakes with uncolored eyes, then played dice, then gave them red or blue eyes. So the snakes, like the players, can have experiences prior to the dice being rolled. And yes, no snakes exist before I start. (why is the god wicked? No love for snakes...) I'll update the text to clarify that no snakes exist until the god of snake creation gets to work.
C) "Some time after this process, you befriended a snake who's always wearing shades. You find out how he was created. Given this, what is the probability that he has red eyes?" -- the answer, following again the same global population argument, is ~50%
I think this is a great crystallization of the paradox. In this scenario, it seems like I should believe I have a 1/36 chance of red eyes, and my new friend has a 1/2 chance of red eyes. But my friend has had exactly the same experiences as me, and they reason that the probabilities are reversed.
I would not have guessed that! So in slightly more formal terms:
Then as I understand your method, it goes something like:
(probabilities may differ, this is just illustrative)
However, I don't think the world is deterministic enough for step 2 to work - the objective probability could be 50% or some other value.