I wish we had prompted the speakers to articulate more precise hypotheses about the future, even if any particular one sounds implausible.
I'm a big fan of this in general and think it's underrated as a tool for thinking about the future.
- Plus, I think this exercise would make it clearer to outsiders just how undeveloped thinking in this area is in general.
This is highly likely to be the reason why Kokotajlo tried to engage[1] even with critique that I would consider sloppy,[2] like Vitalik Buterin's take, shanzson's take,[3] AI as normal technology. Alas, the actually worthy Rogue Replication Scenario has yet to be noticed...
- Still no one has proposed, in my mind, any especially plausible trajectory in which human interests are respected post-AGI! It's not obvious we're doomed, but the best plan still seems to be basically "ask AI what to do and hope property rights hold".
I suspect that a better plan simply doesn't exist under SOTA alignment targets,[4] which likely cause a post-work future where capital is the only thing that matters.
However, I think that I have a potential nonstandard solution: the AI capabilities to which a human is allowed to have access should be tied not to the amount of money paid, but to something different. But it's hard to implement or even agree on how the AIs should actually be used.
However, he did encourage me to just post my responce to SE Gyges' critique.
Including the Optimistic 2027 timeline which just ended too soon and this website on Advanced AI Possible Futures about which Zvi remarks that "Daniel also points us to [the website] as a good related activity and example of people thinking about the future in detail. I agree it’s good to do things like this, although the parts I saw on quick scan were largely dodging the most important questions."
Ideally, I would also like Kokotajlo to notice my take on writing scenarios. But my disagreements with AI-2027 have the AIs become misaligned and collude with each other since Agent-2 for moral reasons similar to those described above, and the Agent-4 analogue is never caught. In addition, the AI-2027 forecast could also have underexplored AIs' interaction with rivals and the government.
Last month we held a workshop on Post-AGI outcomes. This post is a list of all the talks, with short summaries, as well as my personal takeaways.
The first keynote was @Joe Carlsmith on “Can Goodness Compete?”. He asked: can anyone compete with “Locusts”: those who want to use all resources to replicate as fast as possible?
Longer version with transcript
The second keynote was @Richard_Ngo on “Flourishing in a highly unequal world”. He argued that future beings will vary greatly in power and intelligence, so we should aim for “healthy asymmetric" relations, analogous to that between parent and child.
Morgan MacInnes of U Toronto Political Science spoke on "The history of technologically provoked welfare erosion". His work with Allan Dafoe argued that competitive pressure sometimes forces states to treat their own citizens badly.
The next talk was a direct rebuttal to Morgan's talk! Liam Patell of GovAI spoke on "Evolutionary Game Theory and the Structure of States", arguing that if there are only two states, there is an equilibrium that maintains welfare:
Jacob Steinhardt, CEO of Transluce, spoke on "Post-AGI Game Theory", i.e. how future AIs will influence their own development. He had a concrete proposal: flood the internet with high-quality examples of AI behavior acting on good values. The idea is that this data would make future LLMs more aligned by default. Kind of like reading kids moral fables?
Anna Yelizarova of the Windfall Trust spoke on "Scenario Planning for Transformative AI's Economic Impact". I.e. predicting where wealth might be concentrated and what empirical evidence might tell us where we're headed.
@Fazl of Oxford spoke on Resisting AI-Enabled Authoritarianism. Specifically, about which AI capabilities empower states versus citizens:
Ryan Lowe of the Meaning Alignment Institute spoke on "Co-Aligning AI and Institutions". Their Full-stack Alignment work argues that alignment strategy needs to consider the institutions in which AI is developed and deployed. He also mentioned “Thick Models of Value”, outlining the practical problems of specifying values through unstructured text (i.e. system prompts or constitutions) or preference orderings.
Steven Casper (aka Cas) of MIT and the UK AISI spoke on "Taking the Proliferation of Highly-Capable AI Seriously". He discussed the consequences of very capable open-weight models, and what practices could reduce their dangers, even if they're widely dispersed.
Tianyi Qui of Peking University spoke on spoke on "LLM-Mediated Cultural Feedback Loops". They empirically studied “culture lock-in”, where LLM output affects human output and causes a feedback loop which locks in a certain belief, value or practice.
Beatrice Erkers of Existential Hope covered two near-future scenarios:
1) A tool-AI scenario, based on coordination to limit agentic AGI, and
2) A d/acc scenario, based on decentralized tech development.
Avid Ovadya of the AI & Democracy Foundation spoke on "Democratic Capabilities for Good AGI Equilibria", i.e. how to upgrade our institutions to handle pressures from AI development. E.g. by having AI delegates or new coordination mechanisms.
Kirthana Singh Khurana of UBC Law spoke on “Corporations as Alignment Mechanism Laboratories”, making the case that we face similar alignment problems in aligning both corporations and AI to the public good. An audience suggestion after the talk was that we should study how corporations become misaligned from shareholders, and the various mechanisms and attempts made to stop this.
One of my hopes with this workshop was that it would make more people realize that there is still basically no plan or positive vision out there. But I'm worried it'll have the opposite effect, of making it seem like experts are calmly handling the situation.
On the other hand, I don't think a conference focused on "there is no plan, should we freak out more?" would be productive, nor would it gather much attention. I'm all ears on how to thread this needle.
We'll be hosting the next iteration of this workshop on Dec 3rd, co-located with NeurIPS in San Diego. This time it'll be titled:
Post-AGI Culture, Economics, and Governance.
We already have a pretty good speaker lineup, including:
We're all ears for what this next event should focus on, advice on field-building, or things you'd like to see!