Joseph Miller's Shortform

Joseph Miller

This is a special post for quick takes by Joseph Miller. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Lots of people are leaving UK AISI because the pay is too low. Often they go to work on evals at other AI safety orgs where they will get paid more money. This is terrible. It's far more valuable to have at least one government agency in the world with top tier AI expertise than it is to have a slightly stronger non-profit eval ecosystem.

A competent department creates unique options for the UK government to pass a pre-deployment testing law better than other countries. Or the government could empower them as a a thoughtful AI regulator that actually understands the risks. And AISI pushes other countries to improve by setting a high bar. It could even be transformed at some point into an international body that is the backbone of an AI treaty.

The jobs are in fact pretty well paid by UK tech standards but people there often have more lucrative options available at AI safety non-profits. Is there any way we can pay them to stay?

2Lukas Finnveden1mo

I certainly agree with the conceptual point, but it seems like I believe in way faster DMR in the size of the field than you seem to believe in. Like, I think a 50/50 gamble to eliminate the field or triple the size seems bad to me, whereas a 50/50 gamble to either double or halve it feels kind of unclear if it's worth it or not. So feels closer to log than linear to me, although I think it's probably somewhat less steeply diminishing than log.

(I would also have thought that the value of a year of the AI safety nonprofit ecosystem is higher than 50B though, so not at all clear I'm disagreeing with your bottom-line. And probably there's some correlation here, where money is more valuable if there's less steeply DMR to the size of the field.)

5Richard_Ngo1mo

What are you doing now?