StanislavKrym's Shortform

StanislavKrym

This is a special post for quick takes by StanislavKrym. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

David Sacks on X is tone-deaf, especially in bold:

I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible.

But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier.

Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want.

Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well.

So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it.

If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.

Were OAI and Anthropic to shut down new training runs due to control issues, the argument would RELY on GDM NOT reaching the frontier in eight months or on GDM joining the totally-not-cartel.

As for the phrase in italics, it implies that China will refuse any agreement, which IMO is THE crux of AI-2040's Plan A being implementable. Does anyone have contacts in China so that we could learn Chinese sentiment?

11gzjw2mo

On China's Zhihu — the country's highest-quality Q&A platform — under discussions about Jacob's resignation and the recent statements by the "AI Big Three" regarding an AI slowdown, there are a few answers about AI alignment. However, most people know nothing about alignment and do not believe in AI risks, instead framing the issue purely in terms of marketing hype, the breakdown of scaling laws, and market confidence. Given that Zhihu is considered the platform with the highest quality of discourse in China's Q&A space, the situation is not optimistic.

Chinese are smart people, and their leaders hate loosing control, but just as anybody else, they will need to have their own HuggingFace moment to change their mind. Maybe in a year or so, and maybe with a far worse warning shot.

My worry is not that they won't update at some point. My worry is that, because of Moloch, and because leaders tend more to be people like Trump than Merkel, cooperation could reveal impossible.

16JennaS2mo

I do think David has some good points, though. OpenAI and Anthropic could’ve started screaming for much stronger national and global regulation of AI much earlier than now - I remember discussion on the RSPv3 announcement post on this, I think. They could’ve tried to throw their weight around to get all the other orgs to cooperate with an industry pause. Their attitude towards AI development has been “move fast and break things,” but when it comes to trying to coordinate a slowdown, suddenly they start mumbling about their hands being tied by antitrust. METR has relied on the goodwill of the labs for access and tokens, and is partly funded by an Anthropic board observer. And there really is a compelling business and regulatory risk case for getting your AIs to not commit crimes! I don’t agree with everything he’s saying. Obviously I want every company to be audited. And I want us to actually try coordinating with China. But if someone doesn’t trust the frontier labs, it can be rather difficult to tell if they actually care about the long term future of humanity, or if they’re just following their local incentive gradient.