Thanks. I think the possible failure mode of this definition is now in the opposite direction: it's possible there will be an AI that provides less than 2x acceleration according to this new definition (it's not super good at the type of tasks humans typically do), but it's so good at mass-producing new RL environments or something else, and that mass-production turns out to so useful, that the existence of this model already kicks off a rapid intelligence explosion. I agree this is not too likely in the short term though, so the new imprecise definition is probably kind of reasonable for now. 

I also haven't found great sources when looking more closely. This seems like a somewhat good source, but still doesn't quantify how many dollars a super PAC needs to spend to buy a vote.

I'm starting to feel skeptical how reasonable/well-defined these capability levels are in the modern paradigm. 

My understanding is that reasoning models' training includes a lot of clever use of other AIs to generate data or to evaluate completions. Could AI companies create similarly capable models from the same budget as their newest reasoning models if their employees' brain run at 2x speed, but they couldn't use earlier AIs for data generation or evaluation? 

I'm really not sure. I think plausibly the current reasoning training paradigm just wouldn't work at all without using AIs in training. So AI companies would need to look for a different paradigm, which might work much less well, which I can easily imagine outweighing the advantage of employees running 2x speed. If that's the case, does that mean that GPT-4.1 or whatever AI they used in the training of the first reasoning model was plausibly already more than 2x-ing AI R&D labor according to this post's definition? I think that really doesn't match the intuition that this post tried to convey, so I think probably the definition should be changed, but I don't know what would be a good definition.

FWIW, I get a bunch of value from reading Buck's and Ryan's public comments here, and I think many people do. It's possible that Buck and Ryan should spend less time commenting because they have high opportunity cost, but I think it would be pretty sad if their commenting moved to private channels.

I'm confused. If Fairshake's $100 million was this influential, to the point that "politicians are advised that crypto is the single most important industry to avoid pissing off", why don't other industries spend similar amounts on super PACs? $100 million is just not that much money. 

It has long been a mystery to me why there isn't more money in politics, but I always thought that the usual argument was that studies show that campaign spending matters surprisingly little, and in particular, super PAC dollars are very not effective at getting votes. 

How strong is the evidence that the crypto industry managed to become very influential through Fairshake?

I'm pretty confused by the conclusion of this post. I was nodding along during the first half of the essay: I myself worry a lot about how I and others will navigate the dilemma of exposure to AI super-persuasion and addiction on one side, and paranoid isolationism on the other. 

But then in the conclusion of the post, you only talk about how people will fall into one of these two traps: isolationist religious communes locking their members in until the end of times. 

I worry more about the other trap: people foolishly exposing themselves to too much AI generated super-stimulus and getting their brain fried. I think much more people will be exposed to various addictive AI generated content than the number of people who have strong enough religious communities that they create an isolationist bubble.

I think it's plausible that the people who expose themselves to all the addictive stuff on the AI-internet will also sooner or later get captured by some isolationist bubble that keeps them locked away from the other competing memes: arguably that's the only stable point. But I worry that these stable points will be worse than the Christian co-ops you describe. 

I imagine an immortal man, in the year 3000, sitting at his computer, not having left his house or having talked to a human in almost a thousand years, talking with his GPT-5.5 based AI girlfriend and scrolling his personalized twitter feed, full of AI generated outrage stories rehashing the culture war fights of his youth. Outside his window, there is a giant billboard advertising "Come on, even if you want to fritter your life away, at least use our better products! At least upgrade your girlfriend to GPT-6!" But his AI girlfriend told him to shutter his window a thousand years ago, so the billboard is to no avail.

This is of course a somewhat exaggerated picture, but I really do believe that one-person isolation bubbles will be more common and more dystopian than the communal isolationism you describe. 

in the year 3000, still teaching that the Earth is 6,000 years old

No, it will be 7000 years old by then.

On the other hand, there is another interesting factor in kings losing power that might be more related to what you are talking about (though I don't think this factor is as important as the threat of revolutions discussed in the previous comment).

My understanding is that part of the story for why kings lost their power is that the majority of people were commoners, so the best writers, artists and philosophers were commoners (or at least not the highest aristocrats), and the kings and the aristocrats read their work, and these writer often argued for more power to the people. The kings and aristocrats sometimes got sincerely convinced, and agreed to relinquish some powers even when it was not absolutely necessary for preempting revolutions.

I think this is somewhat analogous to the story of cultural AI dominance in Gradual Disempowerment: all the most engaging content creators are AIs, humans consume their content, the AIs argue for giving power to AIs, and the humans get convinced. 

I agree this is a real danger, but I think there might be an important difference between the case of kings and the AI future. 

The court of Louis XVI read Voltaire, but I think if there was someone equally witty to Voltaire who also flattered the aristocracy, they would have plausibly liked him more. But the pool of witty people was limited, and Voltaire was far wittier than any of the few pro-aristocrat humorists, so the royal court put up with Voltaire's hostile opinions. 

On the other hand, in a post-AGI future, I think it's plausible that with a small fraction of the resources you can get close to saturating human engagement. Suppose pro-human groups fund 1% of the AIs generating content, and pro-AI groups fund 99%. (For the sake of argument, let's grant the dubious assumption that the majority of economy is controlled by AIs.) I think it's still plausible that the two groups can generate approximately equally engaging content, and if humans find pro-human content more appealing, then that just wins out.

Also, I'm kind of an idealist, and I think part of the reason that Voltaire was successful is that he was just right about a lot of things, parliamentary government really leads to better outcomes than absolute monarchy from the perspective of a more-or-less shared human morality. So I have some hope (though definitely not certainty) that AI content creators competing in a free marketplace of ideas will only convince humanity to voluntarily relinquish power if relinquishing power is actually the right choice.

I don't think that the example of kings losing their powers really supports your thesis here. That wasn't a seamless, subtle process of power slipping away. There was a lot of bloodshed and threat of bloodshed involved.

King Charles I tried to exercise his powers as a real king and go against the Parliament, but the people rebelled and he lost his head. After that, his son managed to restore the monarchy, though he needed to agree to some more restrictions on his powers. After that, James II tried to go against the Parliament again, and got overthrown and replaced by another guy who agreed to relinquish the majority of royal powers. After that, the king still had some limited say, but when he tried to do unpopular taxes in America, the colonies rebelled, and gained independence through a violent revolution. Then next door from England, Louis XVI tried to go against the will of his Assembly, and lost his head. After these, the British Parliament started to politely ask their kings to relinquish the remainder of their powers, and the kings wisely agreed, so their family could keep their nominal rulership, their nice castle, and most importantly, their head.

I think the analogous situation would be AIs violently over-taking some countries, and after that, the other countries bloodlessly surrendering to their AIs. I think this is much closer to the traditional picture of AI takeover than to the picture you are painting in Gradual Disempowerment.

Unfortunately, I don't think that "this is how science works" is really true. Science focuses on having a simple description of the world, while Solomonoff induction focuses on the description of the world plus your place in it, being simple.

This leads to some really weird consequences, which people sometimes refer to as the Solomonoff induction being malign.