ryan_greenblatt's Shortform

ryan_greenblatt

This is a special post for quick takes by ryan_greenblatt. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I recently recorded a podcast with Dwarkesh about the potential for (very) fast AI progress and how misaligned AI takeover might happen.

Our conversation focused a lot on threat models from "reward-seeking" AIs. If you're interested in reading more about this, Alex (who works with me at Redwood) has written in a lot more detail about this threat model here

I also talked about how "mundane" misalignment and underelicitation could doom us. I say more about this here.

(I think there are other important threat models, like AIs ending up with (shared) long-run preferences and deciding to fake alignment based on these preferences. For reference, see here and here.)

Some of people at Redwood wrote up some notes to help me prep:

Why are yours and most other predictions for take-off timelines years long? My personal median for take-off via RSI (from a reliable auto-coder to ASI) is a single day. The primary reasons I expect it to be faster:

The potential for intelligence in modern hardware is extreme: modern computers can reach over 3 GHz, whereas humans neurons can't fire more than 1000 times per second. With enough processing cores and similar strategies for reasoning an AI could potentially think 6 OOMs faster than a human (assuming there are no other major bottlenecks for reason... (read more)

2Philipp Risius3mo

Thanks! I'll have to take your word for it (since it's probably unwise to ask for what exactly would make machine learning more efficient), but it does sound concerning, in favor of faster takeoff speeds.

4ryan_greenblatt3mo

I briefly discussed with Dwarkesh why I'm skeptical AI progress is heavily driven by scaling up spending on human experts labeling/making data. My main argument is that spending on researchers and experiment compute seem much higher. But I didn't say very much in the podcast. More precisely, my view is that if spending on having human experts label/make individual data points were fixed at ~$100 million / year (per company), then AI progress would be <25% slower. We didn't have time to get into everything in this podcast (and some content about this recorded at an earlier point was cut), so I'll spell out my view in a bit more detail here: * Spending directly on data (rather than on R&D about data) isn't growing that fast and isn't that high (relative to spending on researchers and especially compared to spending on experiment compute). * It's important to make a distinction between spending directly on making data and science about better processes for making data. E.g., better data mixes like FineWeb count as R&D (and the person doing the R&D needs almost no understanding of individual sequences). * AI automation seems differentially good at accelerating data generation, such that I think improvements in RL environments have mostly been driven by improved AI (and R&D into making better environments) rather than spending on humans making/labeling data, and I expect this to continue. Like data stuff seems particularly amenable to acceleration from weaker AIs. * Structurally, most of what human data labeling does (though not all!) depends on having generally decent judgment rather than on having more expertise than the AIs being trained. * Transfer without domain-specific labels looks decent in practice. E.g., it doesn't seem like Anthropic is hiring a ton of mathematicians and cyber experts to do data labeling, and the AIs are still good and getting better at these domains. Maybe this depends on having labels in some domain, but so long as AIs can label in d

ryan_greenblatt's Shortform

19

Ω 5

ryan_greenblatt's Shortform

19

Ω 5