Strong upvoted to signal boost, but again note I don't know what I'm talking about.

Note that most of the compute in consumer laptops is in their GPUs not their CPUs, so comparing H100 flops to laptop CPU flops does not work for establishing the extent to which your policy would affect consumer laptops.

What I hear is that the natsec people judge people for using "treaty" in cases like this. Maybe MIRI looked into it and has better info than me; idk.

I hear a lot of discussion of treaties to monitor compute or ban AI or whatever. But the word "treaty" has a specific meaning that often isn't what people mean. Specifically, treaties are a particular kind of international agreement that (in the US) require Senate approval, and there are a lot of other types of international agreement (e.g. executive agreements that are just made by the president). Treaties are generally more serious and harder to withdraw from.

As a total non-expert, "treaty" does actually seem to me like the vibe that e.g. our MIRI friends are going for--they want something analogous to major nuclear arms control agreements, which are usually treaties. So to me, "treaty" seems like a good word for the central example of what MIRI wants, but international agreements that they're very happy about won't necessarily be treaties.

I wouldn't care about this point, except that I've heard that national security experts are prickly about the word "treaty" being used when "international agreement" could have been used instead (and if you talk about treaties they might assume you're using the word ignorantly even if you aren't). My guess is that you should usually conform to their shibboleth here and say "international agreement" when you don't specifically want to talk about treaties.

For some donation opportunities, e.g. political donations, that would be a crime.

Where do you think you've spelled this argument out best? I'm aware of a lot of places where you've made the argument in passing, but I don't know of anywhere where you say it in depth.

My response last time (which also wasn't really in depth; I should maybe try to articulate my position better sometime...) was this:

I agree that the regime where mistakes don't kill you isn't the same as the regime where mistakes do kill you. But it might be similar in the relevant respects. As a trivial example, if you build a machine in America it usually works when you bring it to Australia. I think that arguments at the level of abstraction you've given here don't establish that this is one of the cases where the risk of the generalization failing is high rather than low.

The way I'd think about this is:

Currently, intellectual labor from machine learning researchers costs a lot of compute. A $1M/year ML researcher costs the same as having 30 or so H100s. At the point where you have AGI, you can probably run the equivalent of one ML researcher with substantially less hardware than that. (I'm amortizing, presumably you'll be running your models on multiple chips doing inference on multiple requests simultaneously.) This means that some ways to convert intellectual labor into compute efficiency will be cost-effective when they weren't previously. So I expect that ML will become substantially more labor-intensive and have much more finicky special casing.

Thanks for doing this. This result is helpful. We at Redwood will definitely try out these methods.

You don't feel like "I think the risk of misaligned AI takeover is enormously important." suffices?

I think you've substantially misunderstood what Will is talking about. He's not making a recommendation that people rush through things. He's noting what he believes (and I mostly agree) to be huge weaknesses in the book's argument.

Similarly, he's not saying labs have alignment in the bag. He's just noting holes in the book's arguments that extreme catastrophic misalignment is overwhelmingly likely.

All of this together makes me extremely confused if his real view is basically just "I agree with most of MIRI's policy proposals but I think we shouldn't rush to enact a halt or slowdown tomorrow".

I assume that he disagrees with MIRI's headline policy proposal of banning AI research, in the senses that he thinks it's a poor choice of policy recommendation given tractability and the concern that this proposal might cause bad things to happen (like uneven bans on AI research). I don't know what he thinks of whether it would be good to magically institute the MIRI policy proposal; I think it's fundamentally unclear what hypothetical you're even supposed to consider in order to answer that question.

I summarized my view on MIRI's policy suggestions as "poor", but I definitely think it will be extremely valuable to have the option to slow down AI development in the future.