From my perspective most of orthodox alignment research developed before the deep learning revolution ended up being mostly useless and some was net negative as it enshrined inadequate ontologies. Some alignment researchers seem to me to be stuck in a frame that just doesn't apply well to artifacts produced by contemporary ML.

I would not recommend new aspiring alignment researchers to read the Sequences, Superintelligence, some of MIRI's earlier work or trawl through the alignment content on Arbital despite reading a lot of that myself.

The LW content that seems valuable re large language models were written in the last one to three years, most in the last year.

@janus' "Simulators" is less than a year old.

It's not particularly clear that buying time would be very helpful. The history of AI Safety research has largely failed to transfer across paradigms/frames. If we "buy time" by freezing the current paradigm, we'd get research in the current paradigm. But if the current paradigm is not the final form of existentially dangerous AI, such research may not he particularly valuable.

Something else that complicates buying time even if the current paradigm is the final form of existentially dangerous AI is phenomena that emerge at greater scale. We're just fundamentally unable to empirically study such phenomena until we have models at the appropriate scale.

Rather than lengthening timelines to AGI/strong AI, it seems to me that most of the existential safety comes from having a slow/continuous takeoff that allows us to implement iterative alignment and governance approaches.

Freezing AI development at a particular stage does not seem all that helpful to me.

That could lead to a hardware overhang and move us out of the regime where only a handful of companies can train strong AI systems (with training runs soon reaching billions of dollars) to a world where hundreds or thousands of actors can do so.

Longer timelines to strong AGI plus faster takeoff (as a result of a hardware overhang) seems less safe than shorter timelines to strong AGI plus slower takeoff.

Reply

[-]Cleo Nardo3y92

I believe we should limit AI development to below 0.2 OOMs/year which is slow continuous takeoff.

Reply

[-]DragonGod3y70

Something like that sounds like a sensible proposal to me.

I'm not sure I endorse that as stated (I believe returns to intelligence are strongly sublinear, so a fixed slow rate of scaling may end up taking too long to get to transformative AI for my taste), but I endorse the general idea of deliberate attempts to control AI takeoff at a pace we can deal with (both for technical AI safety and governance/societal response approaches).

I was pushing back against the idea of an indefinite moratorium now while we harvest the gains from developments to date.

That could lead to a hardware overhang and move us out of the regime where only a handful of companies can train strong AI systems to a world where hundreds or thousands of actors can do so.

Reply

[-]TinkerBird3y51

But if the current paradigm is not the final form of existentially dangerous AI, such research may not he particularly valuable.

I think we should figure out how to train puppies before we try to train wolves. It might turn out that very few principles carry over, but if they do, we'll wish we delayed.

The only drawback I see to delaying is that it might cause people to take the issue less seriously than if powerful AI's appear in their lives very suddenly.

Reply

[-]DragonGod3y33

I endorse attempts to deliberately engineer a slow takeoff. I am less enthused about attempts to freeze AI development at a particular level.

Reply

[-]Dan3y30

A pause, followed by few immediate social effects and slower AGI development then expected may make things worse in the long run. Voices of caution may be seen to have 'cried wolf'.

I agree that humanity doesn't seem prepared to do anything very important in 6 months, AI safety wise.

Edited:Clarity.

Reply

[-]Dan3y*-10

I would not recommend new aspiring alignment researchers to read the Sequences, Superintelligence, some of MIRI's earlier work or trawl through the alignment content on Arbital despite reading a lot of that myself.

I think aspiring alignment researchers should read all these things you mention. This all feels extremely premature. We risk throwing out and having to rediscover concepts at every turn. I think Superinelligence, for example, would still be very important to read even if dated in some respects!

We shouldn't assume too much based on our current extrapolations inspired by the systems making headlines today.

GPT-4's creators already want to take things in a very agentic direction, which may yet negate some of the apparent dated-ness.

"Equipping language models with agency and intrinsic motivation is a fascinating and important direction for future research" - ~~OpenAI in~~ Sparks of Artificial General Intelligence: Early experiments with GPT-4.

Reply