There are 4 points of disagreement I have about this post.
First, I think it's fundamentally based on a strawperson.
my fundamental objection is that their specific strategy for delaying AI is not well targeted.
This post provides an argument for not adopting the "neo-luddite" agenda or not directly empowering neo-luddites. This is not an argument against allying with neo-luddites for specific purposes. I don't know of anyone who has actually advocated for the former. This is not how I would characterize Katija's post.
Second, I think there is an inner strawperson with the example about text-to-image models. From a bird's eye view, I agree with caring very little about these models mimicking humans artistic styles. But this is not where the vast majority of tangible harm may be coming from with text-to-image models. I think that this most likely comes from non-consensual deepfakes being easy to use for targeted harassment, humiliation, and blackmail. I know you've seen the EA forum post about this because you commented on it. But I'd be interested in seeing a reply to my reply to your comment on the post.
Third, I think that this post fails to consider how certain (most?) regulations that neo-luddites would support could meaningfully slow risky things down. In general, any type of regulation that makes research and dev for risky AI technologies harder or less incentivized will in fact slow risky AI progress down. I think that the one example you bring up--text-to-image models--is a counterexample to your point. Suppose we pass a bunch of restrictive IP laws that make it more painful to research, develop, and deploy text-to-image models. That would suddenly slow down this branch of research which could concievably be useful for making riskier AI in the future (e.g. multimodal media generators), hinder revenue opportunities for companies who are speeding up risky AI progress, close off this revenue option to possible future companies who may do the same, and establish law/case law/precedent around generative models that could be set precedent or be repurposed for other types of AI later.
Fourth, I also am not convinced by the specific argument about how indiscriminate regulation could make alignment harder.
Suppose the neo-luddites succeed, and the US congress overhauls copyright law. A plausible consequence is that commercial AI models will only be allowed to be trained on data that was licensed very permissively, such as data that's in the public domain...Right now, if an AI org needs some data that they think will help with alignment, they can generally obtain it, unless that data is private.
This is a nitpick, but I don't actually predict this scenario would pan out. I don't think we'd realistically overhaul copyright law and have the kind of regime with datasets that you describe. But this is probably a question for policy people. There are also neo-luddite solutions that your argument would not apply to--like having legal requirements for companies to make their models "forget" certain content upon request. This would only be a hindrance to the deployer.
Ultimately though, what matters is not whether something makes certain alignment research harder. It matters how much something makes alignment research harder relative to how much it makes risky research harder. Alignment researchers are definitely the ones that are differentially data-hungry. What's a concrete, concievable story in which something like the hypothetical law you described makes things differentially harder for alignment researchers compared to capabilities researchers?
To summarize,
It appears we are in the midst of a new wave of neo-luddite sentiment.
Earlier this month, digital artists staged a mass protest against AI art on ArtStation. A few people are reportedly already getting together to hire a lobbyist to advocate more restrictive IP laws around AI generated content. And anecdotally, I've seen numerous large threads on Twitter in which people criticize the users and creators of AI art.
Personally, this sentiment disappoints me. While I sympathize with the artists who will lose their income, I'm not persuaded by the general argument. The value we could get from nearly free, personalized entertainment would be truly massive. In my opinion, it would be a shame if humanity never allowed that value to be unlocked, or restricted its proliferation severely.
I expect most LessWrong readers to agree with me on this point — that it is not worth sacrificing a technologically richer world just to protect workers from losing their income. Yet there is a related view that I have recently heard some of my friends endorse: that nonetheless, it is worth aligning with neo-luddites, incidentally, in order to slow down AI capabilities.
On the most basic level, I think this argument makes some sense. If aligning with neo-luddites simply means saying "I agree with delaying AI, but not for that reason" then I would not be very concerned. As it happens, I am sympathetic to many of the arguments in Katja Grace's recent post about delaying AI in order to ensure existential AI safety.
Yet I worry that some people intend their alliance with neo-luddites to extend much further than this shallow rejoinder. I am concerned that people might work with neo-luddites to advance their specific policies, and particular means of achieving them, in the hopes that it's "better than nothing" and might give us more time to solve alignment.
In addition to possibly being mildly dishonest, I'm quite worried such an alliance will be counterproductive on separate, purely consequentialist grounds.
If we think of AI progress as a single variable that we can either accelerate or decelerate, with other variables held constant upon intervention, then I agree it could be true that we should do whatever we can to impede the march of progress in the field, no matter what that might look like. Delaying AI gives us more time to reflect, debate, and experiment, which prima facie, I agree, is a good thing.
A better model, however, is that there are many factor inputs to AI development. To name the main ones: compute, data, and algorithmic progress. To the extent we block only one avenue of progress, the others will continue. Whether that's good depends critically on the details: what's being blocked, what isn't, and how.
One consideration, which has been pointed out by many before, is that blocking one avenue of progress may lead to an "overhang" in which the sudden release of restrictions leads to rapid, discontinuous progress, which is highly likely to increase total AI risk.
But an overhang is not my main reason for cautioning against an alliance with neo-luddites. Rather, my fundamental objection is that their specific strategy for delaying AI is not well targeted. Aligning with neo-luddites won't necessarily slow down the parts of AI development that we care about, except by coincidence. Instead of aiming simply to slow down AI, we should care more about ensuring favorable differential technological development.
Why? Because the constraints on AI development shape the type of AI we get, and some types of AIs are easier to align than others. A world that restricts compute will end up with different AGI than a world that restricts data. While some constraints are out of our control — such as the difficulty of finding certain algorithms — other constraints aren't. Therefore, it's critical that we craft these constraints carefully, to ensure the trajectory of AI development goes well.
Passing subpar regulations now — the type of regulations not explicitly designed to provide favorable differential technological progress — might lock us into bad regime. If later we determine that other, better targeted regulations would have been vastly better, it could be very difficult to switch our regulatory structure to adjust. Choosing the right regulatory structure to begin with likely allows for greater choice than switching to a different regulatory structure after one has already been established.
Even worse, the subpar regulations could even make AI harder to align.
Suppose the neo-luddites succeed, and the US congress overhauls copyright law. A plausible consequence is that commercial AI models will only be allowed to be trained on data that was licensed very permissively, such as data that's in the public domain.
What would AI look like if it were only allowed to learn from data in the public domain? Perhaps interacting with it might feel like interacting with someone from a different era — a person from over 95 years ago, whose copyrights have now expired. That's probably not the only consequence, though.
Right now, if an AI org needs some data that they think will help with alignment, they can generally obtain it, unless that data is private. Under a different, highly restrictive copyright regime, this fact may no longer be true.
If deep learning architectures are marble, data is the sculptor. Restricting what data we're allowed to train on shrinks our search space over programs, carving out which parts of the space we're allowed to explore, and which parts we're not. And it seems abstractly important to ensure our search space is not carved up arbitrarily — in a process explicitly intended for unfavorable ends — even if we can't know now which data might be helpful to use, and which data won't be.
True, if very powerful AI is coming very soon (<5 years from now), there might not be much else we can do except for aligning with vaguely friendly groups, and helping them pass poorly designed regulations. It would be desperate, but sensible. If that's your objection to my argument, then I sympathize with you, though I'm a bit more optimistic about how much time we have left on the clock.
If very powerful AI is more than 5 years away, we will likely get other chances to get people to regulate AI from a perspective we sympathize with. Human extinction is actually quite a natural thing to care about. Getting people to regulate AI for that explicit reason just seems like a much better, and more transparent strategy. And while AI gets more advanced, I expect this possibility will become more salient in people's minds anyway.