Why should we think that the relevant progress driving non-formal IMO is very important for plausibly important capabilities like agentic software engineering? [...] if the main breakthrough was in better performance on non-trivial-to-verify tasks (as various posts from OpenAI people claim), then even if this generalizes well beyond proofs this wouldn't obviously particularly help with agentic software engineering (where the core blocker doesn't appear to be verification difficulty).

I'm surprised by this. To me it seems hugely important how fast AIs are improving on tasks with poor feedback loops, because obviously they're in a much better position to improve on easy-to-verify tasks, so "tasks with poor feedback loops" seem pretty likely to be the bottleneck to an intelligence explosion.

So I definitely do think that "better performance on non-trivial-to-verify tasks" are very important for some "plausibly important capabilities". Including agentic software engineering. (Like: This also seems related to why the AIs are much better at benchmarks than at helping people out with their day-to-day work.)

Thanks for the post!

One aspect of "night watchman as transition" is that the negotiated exit from the night watchman state might depend a lot on what's the BATNA that happens in the absence of such an exit. Today, geopolitics is shaped by the fact that war is available as a last resort. If war is banned, then it matters a lot what this new "last resort" is.

This seems strongly related to the section on "Preventing premature claims to space". It seems probably reasonable to specify the BATNA on Earth as whatever you'd get with present property rights + no war. But it seems hard to do something similar for unclaimed resources, like in space.

And for any particular new proposed BATNA, major powers might object to that version of the night watchman if they think it leaves them worse-off than the default trajectory. If the gains from trade are large enough, it might well be possible to find a compromise that people would agree to. But there's a risk that it it requires you to decide on a bunch of thorny questions up-front, rather than being able to kick the can down the road.

If they did, would it work? No.

If an overwhelming majority of civil society plus the USG was pressuring OpenAI in this direction, I think it would have a substantial effect. If only a few non-profits did it, I think it would have little effect. 

To make your analogy work, we need to tell whether the relationship between OpenBrain employees and their AIs is more like "USG + civil society vs. OpenAI" or more like "a few non-profits vs. OpenAI". I'd say "OpenBrain vs. their AIs" is more like "USG + civil society vs. OpenAI". So if all of OpenBrain is on board with d/acc and doing the thing Tom said, I think it would have a substantial effect on the AIs.

And again, you don't need to offer a large % of all future resources if the AI has DMR in resources. (I agree it's a lot harder to credibly offer a large fraction of the stars.)

Yeah, agreed. (That's why I specified "resource hungry" in my original message.)

Makes sense. Though flagging this is then a dimension on which humans can realistically get potentially better placed than AIs. They can rely more on legal institutions as well as trying to engineer situations with joint bargaining power. (Though again, perhaps you'll say AIs will be more willing than humans to actually engineer those situations, which does seem right to me.)

Yeah. Also, I think it'd be hard to engineer significant joint bargaining power (not reliant on anyone's good intentions) without having some government on board.

• Difficult for a few individuals to give AI legal rights that humans are unlikely to reverse.
• Difficult for a few individuals to give AI weapons that would let them impose big costs on humans in the future.

Though if the AIs have big DMR then maybe they're happy with a big bitcoin wallet or something.

Your analysis is focused on whether humans or misaligned AI are at an overall better position at giving out certain deals. But even if I condition on it "humans could avoid AI takeover by credibly offering AIs large percentages of all future resources", it still seems <50% likely that they do it. Curious if you disagree. (In general, if I thought humans were going to act rationally and competently to prevent AI takeover risk, I think that would cut the risk in significantly more than half. There's tons of stuff that we could do to reduce the risk that I doubt we'll do.)

Maybe there's some argument along the lines of "just like humans are likely to mess up in their attempts to prevent AI takeover risk (like failing to offer deals), AIs are likely to mess up in their attempts to take over (like failing to make deals with each other), so this doesn't cut asymmetrically towards making deals-between-AIs more likely". Maybe, I haven't though much about this argument. My first-pass answer would be "we'll just keep making them smarter until they stop messing up".

If you wrote a vignette like Daniel suggests, where humans do end up making deals, that might help me feel like it's more intuitively likely to happen.

Minor points:

It'll be explicitly aiming to break the law and lie to humans to seize power, making its "promises" to other AIs less credible.

I'm generally thinking that the AIs would try to engineer some situations where they all have some bargaining power after the take-over, rather than relying on each others' promises. If you could establish that's very difficult to do, that'd make me think the "coordinated takeover" seemed meaningfully less likely.

Seems v plausible, but why 'probably'? Are you thinking techniques like debate probably stop working?

Yes, because of known issues like inaccessible information (primarily) and obfuscated arguments (secondarily).

I'm in favor of trying to offer deals with the AIs.

I don't think it reliably prevents AI takeover. The situation looks pretty rough if the AIs are far smarter than humans, widely deployed, and resource-hungry. Because:

• It's pretty likely that they'll be able to communicate with each other through one route or another.
• It seems intuitively unlikely that humans will credibly offer AIs large percentages of all future resources. (And if an argument for hope relies on us doing that, I think that should be clearly flagged, because that's still a significant loss of longtermist value.)
• At some level of AI capability, we would probably be unable to adjudicate arguments about which factions are misaligned or about what technical proposals would actually leave us in charge vs. disempowered.

I agree it's not a slam dunk.

It does seem unlikely to me that humanity would credibly offer large fractions of all future resources. (So I wouldn't put it in a scenario forecast meant to represent one of my top few most likely scenarios.)

Importantly, if there are multiple misaligned superintelligences, and no aligned superintelligence, it seems likely that they will be motivated and capable to coordinate with each other to overthrow humanity and divide the spoils.

I think the argument against that the military thing is supposed to be item 1 on the list.

(1) The world's physical security (incl bio and anti-drone) is run by localized authority (whether human jor AI) that is not all puppets of Consensus-1 (the name for the AI that ends up controlling the world and then killing everyone in the AI 2027 scenario) (...) Intuitively, (1) could go both ways. Today, some police forces are highly centralized with strong national command structures, and other police forces are localized. If physical security has to rapidly transform to meet the needs of the AI era, then the landscape will reset entirely, and the new outcomes will depend on choices made over the next few years. Governments could get lazy and all depend on Palantir. Or they could actively choose some option that combines locally developed and open-source technology. Here, I think that we need to just make the right choice.

I.e.: The argument is that there might not be a single Consensus-1 controlled military even in the US.

I think it seems unlikely that the combined US AI police forces will be able to compete with the US AI national military, which is one reason I'm skeptical of this. Still, if "multiple independent militaries" would solve the problem, we could potentially push for that happening inside the national military. It seems plausible to me that the government will want multiple companies to produce AI for their military systems, so we could well end up with different AI military units run by different AI system.

The more fundamental problem is that, even if the different AIs have entirely different development histories, they may all end up misaligned. And if they all end up misaligned, they may collude to overthrow humanity and divide the spoils.

I'm all for attempts to make this more difficult. (That's the kind of thing that the AI control agenda is trying to do.) But as the AIs get more and more superhuman, it starts to seem extremely hard to prevent all their opportunities at collusion.