There's also a more obscure skill, perhaps only useful for Newcomb-like problems.
Asking "what happened before?"
For example "if I observe myself not intending to pay the Parfit's Hitchhiker, how is that compatible with him saving me beforehand?"
You can try applying it yourself before looking at the spoiler.
spoiler!
In your position, what was Black's previous move?
Lmao you caught me. The position was actually taken from here: https://www.chessprogramming.org/Perft_Results
It's a list of positions used to test chess engines, and they tend to be kinda complicated, but yeah this position doesn't seem to actually be legally achievable.
The recurring idea here, about what happens after superintelligence, is that it sets out to reshape the entire universe ("the future light-cone") according to whatever its values happen to be.
In the game "The choice before us" by Nick Shapiro,[1] you are put in the shoes of an AI company leader. You grow your business. You unlock "wonders", such as curing cancer. All the while, you're attempting to avoid your product getting smart enough to escape and take over. You win by achieving 5 wonders without unleashing uncontrolled AI.
I love this game, but it has the major flaw that when you win, you are normally very close to superintelligence. What happens afterwards? You turn the GPUs off? Go home? Get some sleep? The game seems to think so.
This failure to ask "What happens next?" seems to be a broader phenomenon within the AI community. It was in fact the sole question I needed to ask a capabilities researcher for them to take the threat of superintelligence seriously. It's my main weapon against people claiming there are many possible worlds "where only 90% of people die" (if a rogue AI has gone off the rails and killed 90% of your population, you probably no longer have control of the planet, and I have little faith in the survival of everybody else). More broadly, I just wish people would ask this question more often.
"But Sean!" you say. "I cannot keep asking what comes next forever. I'll end up wondering what happens after the pope becomes US president in the year 2124."
And you would be correct! You cannot, in fact, keep on doing this forever. The tree of possibilities is infinite, and spending your life exploring them is reserved for the brave of heart, the noble of mind, and those who have nothing more productive to be doing. We would be better off finding a place to stop.
To work out how, let's look at a simpler version. In particular, the game of chess. More specifically, this position:
In case you're wondering, this position is a mess
For the uninitiated, this is a complex position. The white king is threatened. The black queen is under attack. So is the white rook, a white bishop and one of the white knights. More broadly, the arrows in the image show the main pieces of tension that one would want to resolve before attempting any sort of evaluation; it's no use counting up the pieces if you lose your queen next turn!
In chess engine parlance, we would like to perform a "quiescence search". This search goes only through the moves which resolve tension, arriving at a position like the following:[2]
With all the mess cleared, we can now see that black is up a bishop and a knight and will probably win the game.
The point of this exercise? When evaluating the future, only evaluate futures where things are relatively stationary. The world will be changing, naturally, but there is a difference between worlds where Anthropic takes over the US government and worlds where capabilities plateau. (Oh, I'm sorry, did you think that a government takeover was a straightforward lock-in scenario? When there is no precedent for it in US history? When this is flipping the power structure of the entire globe?). Don't think your way halfway into the singularity and declare that we're in a good position. That's how you lose your queen. Get to a stable point. They're safer.
Which is great and you should try out by the way
After the moves 1. Kh1 Qxb4 2. Rb1 gxh6 3. d3 Bxa7