I am interviewing AI experts on what they think will happen with AI. Here is Katja Grace and her thoughts. AI risk scares me but often I feel pretty disconnected from it. This has helped me think about it.

Here are Katja’s thoughts in brief:

Will we get AI tools that can run a civil service department with 100x fewer people by 2040? 50%
In what fraction of worlds are the majority of AI tools controlled by agentic AI? 70%
In what fraction of worlds will agentic AI achieve their goals? 90%
In what fraction of agentic worlds will the majority of AI controlled resources be implemented for bad goals? 50%
Will policy slow AGI down by 10 years? 25%, How much reduction of badness does 10 years buy us? 30%

You can look at an interactive graph here:

or see all the graphs I’ve done so far https://estimaker.app/ai. There is an image at the bottom of this page.

You can watch the full video here:

Longer explanations

Please remember this is by 2040. This was originally a table

Question
- Katja's Commentary
- My commentary
Will we get AI tools that can run a civil service department with 100x fewer people by 2040? (50%)
- Katja's commentary: Katja says she’s unsure and gives 50% accordingly.
- My commentary:
  - A common failure mode in my discussions about AGI is how vague the capabilities that make “AGI” are. Some people are envisaging tools capable of flawlessly running worldwide conspiracies while others seem to think of top human level coding ability.
  - Katja and I settled on this example because it involves a varied set of tasks at scale and replacing a lot of human labour.
  - The tasks include:
    - Large scale engineering projects
    - Cash transfers
    - Real world disagreements (how to set tax rates, etc)
    - Complex system problems (traffic flows, healthcare takeup)
  - Civil Services act on a scale of millions of people.
In what fraction of those worlds does agentic ever control the majority of resources (even via AI tools) (70%) If you add up the resources controlled by agentic AI and the AI tools they control, is that more than half?
- Katja's commentary:
  - Katja’s intuition is that for extinction, agency is probably necessary.
  - “The reason I think that some of it is probably agentic is that it seems like agents or something roughly agent-like are economically valuable. People want to make things like agents. They are already trying to make things like agents.”
  - “I

...

AI Probability Trees