This is a linkpost for https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
You can think of it as “dangerous capabilities in everyone’s hands”, but I prefer to think of it as “everyone in the world can work on alignment in a hands-on way, and millions of people are exposed to the problem in a much more intuitive and real way than we ever foresaw”.
Ordinary people without PhDs are learning what capabilities and limitations LLMs have. They are learning what capabilities you can and cannot trust an LLM with. They are coming up with creative jailbreaks we never thought of. And they’re doing so with toy models that don’t have superhuman powers of reasoning, and don’t pose X-risks.
It was always hubris to think only a small sect of people in the SF bay area could be trusted with the reins to AI. I’ve never been one to bet against human ingenuity, and I’m not about to bet against them now that I’ve seen the open source community use LLaMa to blaze past every tech company.
I guess where we differ is I don't believe we get an equilibrium.
I believe that the offense-defense balance will strongly favour the attacker, such that one malicious actor could cause vast damage. One illustrative example: it doesn't matter how many security holes you patch, an attacker only needs one to compromise your system. A second: it is much easier to cause a pandemic than to defend against one.
This (purported) leak is incredibly scary if you're worried about AI risk due to AI capabilities developing too fast.
Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.
Secondly, it suggests that data size is less important than you might think from recent scaling laws, as you can achieve high performance with "small, highly curated datasets". This suggests a world where dangerous capabilities are much further distributed.
Thirdly, it proposes that Google should open-source its AI technology. Obviously, this is only the opinion of one researcher and it currently seems unlikely to occur, but if Google was to pursue this path, this would lead to a significant shortening of timelines.
I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.