I think on the object level, one of the ways I'd see this line of argument falling flat is this part

Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to).

I am not at all comfortable relying on nobody deploying just because there are obvious legible problems. With the right incentives and selection pressures, I think people can be amazing at not noticing or understanding obvious understandable problems. Actual illegibility does not seem required.

In my experience, the main issue with this kind of thing is finding really central examples of symmetries in the input that are emulatable. There's a couple easy ones, like low rank^[1] structure, but I never really managed to get a good argument for why generic symmetries in the data would often be emulatable^[2] in real life.^[3]

You might want to chat with Owen Lewis about this. He's been thinking about connections between input symmetries and mechanistic structure for a while, and was interested in figuring out some kind of general correspondence between input symmetries and parameter symmetries.

1. ^{^}

   If $q\left(x\right)$ only depends on a low-rank subspace of the inputs $x$, there will usually^[4] be degrees of freedom in the weights that connect to that input vector. The same is true of the hidden activations, if they're low rank, we get a corresponding number of free weights. See e.g. section 3.1.2 here.

2. ^{^}

   Good name for this concept by the way, thanks.

3. ^{^}

   For a while I was hoping that almost any kind of input symmetry would tend to correspond to low-rank structure in the hidden representations of $p\left(x\right|{\Theta }^{\ast })$, if $p(.)$ has the sort of architecture used by modern neural networks. Then, almost any kind of symmetry would be reducible to the low-rank structure case^[2], and hence almost any symmetry would be emulatable. 
   
   But I never managed to show this, and I no longer think it is true.

4. ^{^}

   There are a couple of necessary conditions for this of course. E.g. the architecture $p(.)$ needs to actually use weight matrices, like neural networks do.

The WaPo article appears to refer to passenger fatalities per billion passenger miles, not total fatalities. For comparison, trains in the European Union in 2021 apparently had ca. 0.03 passenger fatalities per billion passenger miles, but almost 0.3 total fatalities per million train miles.

Right now it reads like one example of the pledged funding being met, one example of it being only being ca. 3/4 met but there's also two years left until the original deadline, and one example of the funding never getting pledged in the first place (since congress didn't pass it).

I agree this is a pitifully small investment. But it doesn't seem like big bills and programs got created and then walked back. More like they just never came to be in the first place. 4.5 billion euros is a paltry sum.

I think this may be an important distinction to make, because it suggests there was perhaps never much political push to prepare for the next pandemic even at the time. Did people actually 'memory hole' and forget, or did they just never care in the first place?

I for one don't recall much discussion about preparing for the next pandemic outside rationalist/EA-adjacent circles even while the Covid-19 pandemic was still in full swing.

The Pandemic Fund got pledged $3 bio.
...
the Pandemic Fund has received $3.1 bio, with an unmet funding gap of $1 bio. as of the time of writing.

I'm confused. This makes it sound like they did get the pledged funding?

For what it's worth, my mother read If Anyone Builds It, Everyone Dies and seems to have been convinced by it. She's probably not very representative though. She had prior exposure to AI x-risk arguments through me, is autistic, has a math PhD, and is a Gödel, Escher, Bach fan. 

The proposal at the end looks somewhat promising to me on a first skim. Are there known counterpoints for it?

I agree that this seems maybe useful for some things, but not for the "Which UTM?" question in the context of debates about Solomonoff induction specifically, and I think that's the "Which UTM?" question we are actually kind of philosophically confused about. I don't think we are philosophically confused about which UTM to use in the context of us already knowing some physics and wanting to incorporate that knowledge into the UTM pick, we're confused about how to pick if we don't have any information at all yet.

Attempted abstraction and generalization: If we don't know what the ideal UTM is, we can start with some arbitrary UTM $U_1$, and use it to predict the world for a while. After (we think) we've gotten most of our prediction mistakes out of the way, we can then look at our current posterior, and ask which other UTM $U_2$ might have updated to that posterior faster, using less bits of observation about (our universe/the string we're predicting). You could think of this as a way to define what the 'correct' UTM is. But I don't find that definition very satisfying, because the validity of this procedure for finding a good $U_2$ depends on how correct the posterior we've converged on with our previous, arbitrary, $U_1$ is. 'The best UTM is the one that figures out the right answer the fastest' is true, but not very useful.

Is the thermodynamics angle gaining us any more than that for defining the 'correct' choice of UTM? 

We used some general reasoning procedures to figure out some laws of physics and stuff about our universe. Now we're basically asking what other general reasoning procedures might figure out stuff about our universe as fast or faster, conditional on our current understanding of our universe being correct. 

Why does it make Bayesian model comparison harder? Wouldn't you get explicit predicted probabilities for the data $X$ from any two models you train this way? I guess you do need to sample from the Gaussian in $\lambda$ a few times for each $X$ and pass the result through the flow models, but that shouldn't be too expensive.