The following contains resources that I (Eleni) curated to help the AI Science team of AI Safety Camp 2023 prepare for the second half of the project, i.e., forecasting science capabilities. Suggestions for improvement of this guide are welcome.
What is a hard take off?: “A hard takeoff (or an AI going "FOOM"[2]) refers to AGI expansion in a matter of minutes, days, or months. It is a fast, abruptly, local increase in capability. This scenario is widely considered much more precarious, as this involves an AGI rapidly ascending in power without human control. This may result in unexpected or undesired behavior (i.e.Unfriendly AI). It is one of the main ideas supporting theIntelligence explosion hypothesis.
What is a soft take off?: “A soft takeoff refers to an AGI that would self-improve over a period of years or decades. This could be due to either the learning algorithm being too demanding for the hardware or because the AI relies on experiencing feedback from the real-world that would have to be played out in real time.”
What is a sharp left turn?: the transition from a slower to a faster scaling regime as defined here (6:27 to 7:50).
Pivotal acts: acts that we, humans, take that make a big difference in terms of making x-risk less likely.
Objection: AI alignment doesn’t seem like a purely mathematical problem - it requires knowledge about different aspects of reality that we haven’t been able to formalize (yet) e.g., how do agents (both human and artificial) represent values.
Follow-up objection: if we knew how to formalize these problems then we wouldn’t need much help from AI science models, i.e., if we knew what questions to ask and how to ask them.
Such questions would be: “can you solve this [input difficult for humans yet solvable mathematical puzzle e.g., a matrix multiplication]" it . (see below in STEM AGI).
How far can we predict?:
According to the Event Horizon Thesis, whatever happens once we have superintelligence is unpredictable and can’t compare to other technological advances we have already observed/studied.
How much detail can we have?: If the Event Horizon Thesis is correct, detailed predictions probably stop right before the singularity.
Aim for mechanistic explanations for any claims or prediction, meaning always offer a detailed account of how a process or system works in terms of its components, their interactions, and any underlying principles governing their behavior.
There’s a lot to say about explanation; you might say, well, we’re trying to predict, not to explain. To explain is more like the opposite of predicting ( you may have read that explanation = retrodiction). The two processes have a lot in common but what I’m trying to say is stick to the details!
If you’d like to read some posts to inform your thinking about explanation, I highly recommend:
Even worse: probably something involving s-risks (that I thankfully couldn't find). For ideas on what this could mean, see Superintelligence “Would maximally efficient work be fun?”
Would a neutral end make sense? What would that even mean? E.g., there is superintelligence but leaves us alone.
Understanding STEM AGI scenarios
The main idea is in STEM AGI i.e., train science models in formal language, not natural language, keep them in a box and, make sure they have no cognitive/epistemic tools to develop situational awareness.
But also, we want these models to be really good at tackling complex scientific tasks while not being autonomous in any harmful ways.
It seems strange to want to have effective solutions for problems of the physical world without any input from the physical world.
In which world do we live?
Looks like we are in a world where the exact opposite is happening: we mostly have very successful NLP models.
GPT-x models have already read the internet many times and probably have enough informationto at least simulate the human world in different degrees of detail, depending on the prompt.
We don’t exactly know what GPT-x models know, but here are some hypotheses:
The model operates according to an 1-1 correspondence with the external world through information it retrieves from the internet. Each token is a representation of an object/concept in the human/natural world.
The model is a simulation of the external world. This simulation is like the world of a board game: it features properties of the external world in the form of inferences from human text the model has been trained on, including agentic and non-agentic entities, e.g., you’re playing a WW2 game and you’re in the position of the Allies while your friend plays in the position of Germany. The rules of the game are not the rules of the war that actually took place, but rather what the model inferred about the rules from the training dataset.
Current AI models write well enough to be published in scientific journals.
The question is not “when will we have AIs that can trick reviewers into thinking that a paper is good” (that’s a measure of the reviewer’s stupidity); we’re looking for when can the model do science.
When can an AI system publish a paper that satisfies the criteria of “good science” which could entail e.g., lots of citations?
By the time that happens, it’s already too late for alignment.
Conor Leahy predicts
1) that AIs will be able to publish correct science before they can load dishwashers.
2) the world ends before more than 10% of cars on the street are autonomous.
It seems strange to want to have effective solutions for problems of the physical world without any input from the physical world.
To human intuition, yes, but fundamentally, no, not if the needed input is implicit in the training data. Or if the AI can do enough pure math to find better ways to do physical simulation to extrapolate from the data it already has. You don't need to be able to predict General Relativity from three frames of a falling apple to think that maybe, if you have simultaneous access to the results of every paper and experiment ever published, and the specs of every product and instrument and device ever made, that you could reliably come up with some importantly useful new things.
As a minor example, we already live in the world (and have for a decade or more) where materials informatics and computational chemistry tools, with no AGI at all, can cut the number of experiments a company needs to develop a new material in half. Or where generative design software, with no AGI, run by someone not an expert in the relevant field of engineering, can greatly improve the performance of mechanical parts, which can be immediately made with 3D printers. That's a very small subset of STEM for sure, but it exists today.
The following contains resources that I (Eleni) curated to help the AI Science team of AI Safety Camp 2023 prepare for the second half of the project, i.e., forecasting science capabilities. Suggestions for improvement of this guide are welcome.
Key points and readings
for forecasting in general:
for forecasting AI in particular:
If you’d like to read some posts to inform your thinking about explanation, I highly recommend:
Specific changes to consider:
How does change happen?
Scaling Laws: more high-quality data get us better results than more parameters.
Concrete AI stories
Understanding STEM AGI scenarios
Forecasting AGI and science models overlap
1) that AIs will be able to publish correct science before they can load dishwashers.
2) the world ends before more than 10% of cars on the street are autonomous.
Continuity vs discontinuity in AI progress
Meta-scientific considerations