tlevin's Shortform

tlevin

This is a special post for quick takes by tlevin. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

[reposting from Twitter, lightly edited/reformatted] Sometimes I think the whole policy framework for reducing catastrophic risks from AI boils down to two core requirements -- transparency and security -- for models capable of dramatically accelerating R&D.

If you have a model that could lead to general capabilities much stronger than human-level within, say, 12 months, by significantly improving subsequent training runs, the public and scientific community have a right to know this exists and to see at least a redacted safety case; and external researchers need to have some degree of red-teaming access. Probably various other forms of transparency would be useful too. It feels like this is a category of ask that should unite the "safety," "ethics," and "accelerationist" communities?

And the flip side is that it's very important that a model capable of kicking off that kind of dash to superhuman capabilities not get stolen/exfiltrated, such that you don't wind up with multiple actors facing enormous competitive tradeoffs to rush through this process.

These have some tradeoffs, especially as you approach AGI -- e.g. if you develop a system that can do 99% of foundation model training tasks and your security is terrible you do have some good reasons not to immediately announce it -- but not if we make progress on either of these before then, IMO. What the Pareto Frontier of transparency and security looks like, and where we should land on that curve, seems like a very important research agenda.

If you're interested in moving the ball forward on either of these, my colleagues and I would love to see your proposal and might fund you to work on it!

Company/country X has an AI agent that can ~~do 99%~~ [edit: let's say "automate 90%"] of AI R&D tasks, call it Agent-GPT-7, and enough of a compute stock to have that train a significantly better Agent-GPT-8 in 4 months at full speed ahead, which can then train a basically superintelligent Agent-GPT-9 in another 4 months at full speed ahead. (Company/country X doesn't know the exact numbers, but their 80% CI is something like 2-8 months for each step; company/country Y has less info, so their 80% CI is more like 1-16 months for each step.)

Spicy take: it m... (read more)

4ryan_greenblatt2y

I think it should be possible to get a good enough verification regime in practice with considerable effort. It's possible that sufficiently good verification occurs by default due to spies. I agree it there will potentially be a lot of issues downstream of verification issues by default. [...] Hmm, this isn't really how I model the situation with respect to racing. From my perspective, the question isn't "security or no security", but is instead "when will you have extreme security". (My response might overlap with tlevin's, I'm not super sure.) Here's an example way things could go: * An AI lab develops a model that begins to accelerate AI R&D substantially (say 10x) while having weak security. This model was developed primarily for commercial reasons and the possibility of it being stolen isn't a substantial disincentive in practice. * This model is immediately stolen by China. * Shortly after this, USG secures the AI lab. * Now, further AIs will be secure, but to stay ahead of China which has substantially accelerated AI R&D and other AI work, USG races to AIs which are much smarter than humans. In this scenario, if you had extreme security ready to go earlier, then the US would potentially have a larger lead and better negotiating position. I think this probably gets you longer delays prior to qualitatively wildly superhuman AIs in practice. There is a case that if you don't work on extreme security in advance, then there will naturally be a pause to implement this. I'm a bit skeptical of this in practice, especially in short timelines. I also think that the timing of this pause might not be ideal - you'd like to pause when you already have transformative AI rather than before. Separately, if you imagine that USG is rational and at least somewhat aligned, then I think security looks quite good, though I can understand why you wouldn't buy this.

7habryka2y

Interesting, I guess my model is that the default outcome (in the absence of heroic efforts to the contrary) is indeed "no security for nation state attackers", which as far as I can tell is currently the default for practically everything that is developed using modern computing systems. Getting to a point where you can protect something like the weights of an AI model from nation state actors would be extraordinarily difficult and an unprecedented achievement in computer security, which is why I don't expect it to happen (even as many actors would really want it to happen). My model of cybersecurity is extremely offense-dominated for anything that requires internet access or requires thousands of people to have access (both of which I think are quite likely for deployed weights).