This is a special post for quick takes by Raymond Douglas. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Last week we wrapped the second post-AGI workshop; I'm copying across some reflections I put up on twitter:
- The post-AGI question is very interdisciplinary: whether an outcome is truly stable depends not just on economics and shape of future technology but also on things like the nature of human ideological progress and the physics of interplanetary civilizations
- Some concrete takeaways:
- proper global UBI is *enormously* expensive (h/t @yelizarovanna)
- instead of ‘lower costs’, we should talk about relative prices (h/t @akorinek)
- lots human values are actually pretty convergent - they’re shared by many animals (h/t @BerenMillidge)
- Among the many tensions in perspective, one of the more productive ones was between the ‘alignment is easy so let’s try to solve the rest’ crowd and the ‘alignment is hard and maybe this will make people realise they should fully halt AGI’ crowd. Strange bedfellows!
- It’s hard to avoid partisan politics, but part of what’s weird about AGI is that it can upend basic political assumptions. Maybe AGI will outperform the invisible hand of the market! Maybe governments will grow so powerful that revolution is literally impossible!
- Funnily enough, it seems like the main reason people got less doomy was seeing that other people were working hard on the problem, and the main reason people got more doomy was thinking about the problem themselves. Maybe selection effects? Maybe not?
- Compared to last time, even if nobody had good answers to how the world could be nice for humans post-AGI, it felt like we were at least beginning to converge on certain useful perspectives and angles of attack, which seems like a good sign
- Overall, it was a great time! The topic is niche enough that it self-selects a lot for people who actually care, and that is proving to be a very thoughtful and surprisingly diverse crowd. Hopefully soon we’ll be sharing recordings of the talks!
- Bonus: Two other reactions from attendees
- Thanks to all who came, and especially to @DavidDuvenaud, @jankulveit, @StephenLCasper, and Maria Kostylew for organising!
4
1
1
1>proper global UBI is *enormously* expensive (h/t @yelizarovanna)
This seems wrong. There will be huge amounts of wealth post-ASI. Even a relatively small UBI (e.g. 1% of AI companies) will be enough to support way better QOL for everyone on earth. Moreover, everything will become way cheaper because of efficiency gains downstream of AI. Even just at AGI, I think it's plausible that physical labour is something like 10x cheaper and cognitive labour is something like 1000x cheaper.
Sorry! I realise now that this point was a bit unclear. My sense of the expanded claim is something like:
- People sometimes talk about AI UBI/UBC as if it were basically a scaled-up version of the UBI people normally talk about, but it's actually pretty substantially different
- Global UBI right now would be incredibly expensive
- In between now and a functioning global UBI we'd need some mix of massive taxes and massive economic growth (which could indeed just be the latter!)
- But either way, the world in which that happened would not be economics as usual
- (And maybe it is also a huge mess trying to get this set up beforehand so that it's robust to the transition, or afterwards when the people who need it don't have much leverage)
For my part I found this surprising because I hadn't reflected on the sheer orders of magnitude involved, and the fact that any version of this basically involves passing through some fragile craziness. Even if it's small as a proportion of future GDP, it would in absolute terms be tremendously large.
I separately think there was something important to Korinek's claim (which I can't fully regenerate) that the relevant thing isn't really whether stuff is 'cheaper', but rather the prices of all of these goods relative to everything else going on.
2
1I was also there, and my take is there was actually fairly little specific, technical discussion about the economics and politics of what happens post-AGI. This is mostly due to it not being anyone's job to think about these questions, and only somewhat because they're inherently hard questions. Not really sure what I would change.
it seems like the main reason people got less doomy was seeing that other people were working hard on the problem [...]
This would be v surprising to me!
It seems like, to the extend that we're less doomy about survival/flourishing, this isn't bc we've seen a surprising amount of effort, and think effort is v correlated with success. It's more like: our observations increase our confidence that the problem was easy all along, or that we have been living in a 'lucky' world all along.
I might ask you about this when I see you next -- I didn't attend the workshop so maybe I'm just wrong here.
You mean post AGI and pre ASI?
I agree that will be a tricky stretch even if we solve alignment.
Post ASI the only question is whether it's aligned or intent aligned to a good person(s). It takes care of the rest.
One solution is to push fast from AGI to ASI.
With an aligned ASI, other concerns are largely (understandable) failures of the imagination. The possibilities are nearly limitless. You can find something to love.
This is under a benevolent sovereign. The intuitively appealing balances of power seem really tough to stabilize long term or even short term during takeoff.
I'm not at all surprised by the assertion that humans share values with animals. When you consider that selective pressures act on all systems (which is to say that every living system has to engage with the core constraints of visibility, cost, memory, and strain), it's not much of a leap to conclude that there would be shared attractor basins where values converge over evolutionary timescales.
I somehow stumbled upon the 2018-2019 alignment review. Man, it's really hard keeping perspective about how quickly the field is moving. The big signs of AI progress were RL for starcraft and dota, plus some GPT2 variants. The public debates were just starting around continuous vs discontinous takeoff. There are subsections on embedded agency and comprehensive AI services.
I'm not sure what I expected but it's left me feeling a bit humbled.
1I think it’s pretty humbling.
One thing that I’ve been thinking about is how much the landscape will change between now and handoff. If we treat 2019-2026 as 10 “units” of change, then do we get another 10? Or 20? Or 100?
I think I’ve updated towards bigger units. Note that it’s not just the technical landscape that will continue to change (and accelerate, due to R&D automation). It’s also the political landscape, which has been dormant for most of 2019-2026. And soon the geopolitical landscape will start rumbling. And the economic. And the cultural. I think these things are still pretty dormant, compared to how much they will start flipping the overall strategic landscape.
Maybe my median is something like 60-90 units, i.e. we’re about 10-15% of the way through the story, starting at 2019. This makes me much more keen on forward-chainy stuff like “make people more reasonable” and “empower the reasonable people”. (We disagree somewhat on who counts as most reasonable, but I’m imagining a pretty broad class of people which includes both those you think are most reasonable and those I think are most reasonable.)
There’s also an emotional thing which is like… I used to think “wow I’m in such a hingey time in history.” But now I’m like “Q2 2026 isn’t hingey. 2028 is hingey.” And this kinda lowers the stakes a bit?
I like that framing. Somehow intuitively I feel like we're further through the story. Maybe I feel less optimistic about how much good we'll achieve by being reactive as opposed to proactive? I agree 2028 is going to be pretty hingey but I also expect a lot of it might be catch-up for things we're already behind on, like, a lot of the challenge is serial work. And also I feel a bit like we're losing leverage over time.
Some places other than LessWrong where I get useful/complementary AI takes:
- Business/finance newsletters -- mainly Byrne Hobart and Matt Levine. AI is enough of the market now that they both regularly do analyses of the economics, incentives, and politics. As well as keeping up with the news, I feel like this helps me pick up useful intuitions about potentially counterintuitive dynamics -- e.g. why Amazon, despite being a near-monopoly on ecommerce, is pretty good to its customers, and what this might imply about AI. They're also both excellent writers: wry, conversational, and highly information-dense.
- Twitter -- much more psychically draining and conflict-prone, hard for it to be nuanced, but it's also useful to see what the conflicts look like and it feels more like the channels of power so it's good to get a feel for how that works. I get the most mileage out of tracking (1) people who publish papers I like; (2) people whose twitter fights I find it productive to follow [e.g. habryka, ngo, kulveit]; (3) people who aren't on LW much [e.g. janus, brundage, davidad]
- Sentinel -- pretty interesting just seeing how this differs from 'normal news': percentage estimates everywhere, generally focused on which things might blow up into catastrophes. I think the AI news is somewhat dominated by the other things I read these days but I both enjoy the other news and find it interesting to see which AI stuff makes the cut.
- Zvi -- one-stop shop, one man aggregator of all AI news, saves me having to read all of twitter or the model cards. I think this is just the best place for keeping on top of current events. Also I think Zvi is pretty good at pulling out general lessons (although not as much as Byrne or Matt), and very good at the occasional one-off special posts (e.g. his pieces on Claude welfare).
- TRIP -- a general politics podcast which again has been increasingly focused on AI lately as one of the hosts (Rory Stewart) has got more concerned. Most of the concentrated AI juice is in their "Leading" interview series. I found the interview with Jack Clark particularly insightful, because Rory came across as a lot more thoughtful about the risks than Jack.
- Private group chats, slacks etc -- Lots of important information is too sensitive to blast on the public internet where journalists will hunt it down. It is useful to find (or build) the infrastructure for more private discussion.
By contrast, places where I generally feel like I don't get much:
- Newsletters that are doing more like 'pop news', which often feel a bit propaganda-y and watered down to me
- Places that invite contributors to write articles, where I think the average quality of the writing-on-AI is worse than I get from just following a few specific people and a few forums with upvote systems. The exceptions here are probably Asterisk and Works in Progress, although they only sometimes cover AI.
- Governance substackers: not sure why, but I don't feel like I've found any that really click for me. Maybe I'm just not the audience.
- Dwarkesh, which I think is genuinely good, I just so rarely want to watch a two-hour video, which is what most of his recent AI stuff has been, and also I think I disagree with his takes on AI a bit. I do like a lot of his pure interview general interest stuff though. I do appreciate the occasional Zvi summary.
- So much of twitter is a hot mess that's trying to eat your brain
(Epistemic status: trying to get back up to speed on a massive email backlog after a wonderful weekend away.)
On this very day, Matt Levine comments on EA and AI risk:
Now, of course, the way to make real money is by working at a frontier AI lab. This suggests that the optimal current form of effective altruism is:
- Work at a frontier AI lab that is sprinting to build artificial superintelligence as quickly as possible.
- Get paid like $100 million a year.
- Live modestly and donate all of your earnings to a charity that is trying to stop frontier AI labs from building artificial superintelligence that might wipe out humanity.
I can see no flaws in this approach.
I have recently found https://www.lawfaremedia.org/ good for understanding concrete details about current events involving government and the law, including the ones involving AI policy. For example, one author has been writing recently about ways that a coordinated AI pause could be made to be compatible with antitrust law: https://www.lawfaremedia.org/article/can-frontier-ai-labs-lawfully-agree-to-pause
I have found Hard Fork good for weekly roundups on what's happened in AI with commentary from what I'd consider decent journalists (wouldn't bet on them nailing an explanation of how a Transformer works, but in touch enough to be broadly competent in explaining what a current-day AI looks like).
Hacker news consistently surfaces release of less popular models that I almost never see elsewhere naturally, so it could be worth checking https://www.daemonology.net/hn-daily/ occasionally just to find interesting releases that you can then search for. The comments are rarely worth reading even after you dump the whole thing into AI and ask for the best threads. I wouldn't recommend doing this unless you already spent 1+ hour/day looking through AI stuff.