I play on lichess often. I can tell you that a lichess rating of 2900 absolutely corresponds to grandmaster level strength. It is rare for FMs or IMs to exceed a 2800 blitz rating. Most grandmasters hover around 2600-2800 blitz.
The discussion on attack surfaces is very useful, intuitive and accessible. If a better standalone resource doesn’t already exist, such a (perhaps expanded) list/discussion would be a useful intro for people unfamiliar with specific risks.
This was excruciatingly frustrating to read, well done.
This is well-reasoned, but I have difficulty understanding why this kind of takeover would be necessary from the perspective of a powerful, rational agent. Assuming AGI is indeed worth its name, it seems the period of time needed for it to "play nice" would be very brief.
AGI would be expected to be totally unconcerned with being "clean" in a takeover attempt. There would be no need to leave no witnesses, nor avoid rousing opposition. Once you have access to sufficient compute, and enough control over physical resources, why wait 10 years for humanity to be slowly, obliviously strangled?
You say there's "no need" for it to reveal that we are in conflict, but in many cases, concealing a conflict will prevent a wide range of critical, direct moves. The default is a blatant approach - concealing a takeover requires more effort and more time.
The nano-factories thing is a rather extreme version of this, but strategies like poisoning the air/water, building/stealing an army of drones, launching hundreds of nukes, etc., all seem like much more straightforward ways to cripple opposition, even with a relatively weak (99.99th percentile-human-level) AGI.
It could certainly angle for humanity to go out with a whimper, not a bang. But if a bang is quicker, why bother with the charade?
My first thought as well. IGF-1 exists for a reason. Growth is universally necessary for development, repair and function.
shift the
Minor edit - should be "shift in the"
It's encouraging to see more emphasis recently on the political and public-facing aspects of alignment. We are living in a far-better-than-worst-case world where people, including powerful ones, are open to being convinced. They just need to be taught - to have it explained to them intuitively.
It seems cached beliefs produced by works like you get about five words have led to a passive, unspoken attitude among many informed people that attempting to explain anything complicated is futile. It isn't futile. It's just difficult.
In another of your most downvoted posts, you say
I kind of expect this post to be wildly unpopular
I think you may be onto something here.
You can fail to get rid of balls. All of your energy and effort can go into not allowing something to crash or fall, averting each disaster shortly before it would be too late. Speaking for ten minutes with each of fifty of sources every day can be a good way to keep any of them from being completely neglected, but it’s a terrible way to actually finish any of those projects. The terminal stage of this is a system so tied up in maintaining itself and stopping from falling behind that it has no slack to clear tasks or to improve its speed.
This is the salient danger of this approach. While valuable, it absolutely must be paired with a ruthless, exacting and periodic inventory of the balls that matter, otherwise your slack will be completely burned and you will die an exhausted and unaccomplished juggler.
I cannot imagine losing this game as the gatekeeper either, honestly.
Does anyone want to play against me? I’ll bet you $50 USD.