10 quick takes about AGI
8Seth Herd
2Max H
4avturchin
3AnthonyC
1O O
4AnthonyC
7O O
2Max H
1O O
3Max H
1O O
2Max H
1O O
2Max H
1O O
1Noosphere89
2Seth Herd
New Comment
I love the format!
-
agree completely
-
agree completely
-
Capabilities won't be limited by compute OR data? So capabilities will be purely limited by algorithmic improvements? This seems extreme; it seems like better (if not more) data is still useful, and more compute seems highly relevant today.
-
I agree that a human supergenius that just happens to be able to duplicate themselves and think faster with more compute could probably rearrange all the matter in its light cone. A village idiot could not. And I agree that humans are way smarter at some tasks than others. I think this makes discussion of how smart machines can get pretty irrelevant, and it makes the discussion of how fast they can get smarter less relevant. They're almost smart enough to take over, and it will take less than a generation (generously) to get there. We need alignment solutions ASAP. It could already be too late to implement solutions that don't apply to current directions in AI.
-
Agree. Goal direction and abstract reasoning are highly useful for almost any cognitive task. Goal direction allows you to split the task into simpler components and work on them separately. Abstract reasoning saves tons of computation and corresponds to the structure of the world.
-
I think you need to say complex cognition emerges on difficult tasks with enough effort, enough data variety, and good learning algorithms. How much is enough in each category is debatable, and they clearly trade off against each other.
-
Yes, but I don't think evolution is a very useful upper bound on difficulty of thinking up algorithms. Evolution "thinks" completely differently than us or any agent. And it's had an unimaginably vast amount of "compute" in physical systems to work with over time.
-
Yes, but again the comparison is of dubious use. There are a lot of molecules in the brain, and if count each receptor as a calculation (which it is), we'd get a very high compute requirement. It's clearly lower, but how much lower is debatable and debated. I'd rather reason from GPT4 and other existing deep networks directly. GPT4 is AGI without memory, executive function, or sensory systems. Each of those can be estimated from existing systems. My answer would also be about 5x GPT4, except for the likely algorithmic improvements.
-
Agreed, but every time someone says how we "should" build AGI or what we "should" allow it to do, I tune out if it's not followed immediately by an idea of how we COULD convince the world to take this path. Having AI try doing everything is tempting from both an economic and curiosity standpoint.
-
Absolutely. This is the sharp left turn logic, nicely spelled out in A Friendly Face (Another Failure Story). Honesty prior to contextual awareness might be encouraging, but I really don't want to launch any system without strong corrigibility and interpretability. I'm aware we might have to try aligning such a system, but we really should be shooting for better options that align with the practicalities of AI development.
1On (3), I'm more saying, capabilities won't be bottlenecked on more data or compute. Before, say, 2019 (GPT2 release) AI researchers weren't actually using enough data or compute for many potential algorithmic innovations to be relevant, regardless of what was available theoretically at the time.
But now that we're past a minimum threshold of using enough compute and data where lots of things have started working at all, I claim / predict that capabilities researchers will always be able to make meaningful and practical advances just by improving algorithms. More compute and more data could also be helpful, but I consider that to be kind of trivial - you can always get better performance from a Go or Chess engine by letting it run for longer to search deeper in the game tree by brute force.
And it's had an unimaginably vast amount of "compute" in physical systems to work with over time.
A few billion years of very wasteful and inefficient trial-and-error by gradual mutation on a single planet doesn't seem too vast, in the grand scheme of things. Most of the important stuff (in terms of getting to human-level intelligence) probably happened in the last few million years. Maybe it takes planet-scale or even solar-system scale supercomputers running for a few years to reproduce / simulate. I would bet that it doesn't take anything galaxy-scale.
On (9): yeah, I was mainly just pointing out a potentially non-obvious use and purpose of some research that people sometimes don't see the relevance of. Kind of straw, but I think that some people look at e.g. logical decision theory, and say "how the heck am I supposed to build this into an ML model? I can't, therefore this is not relevant."
And one reply is that you don't build it in directly: a smart enough AI system will hit on LDT (or something better) all by itself. We thus want to understand LDT (and other problems in agent foundations) so that we can get out in front of that and see it coming.
What do you think about combinatorial explosion as a possible soft limit to the power of intelligence? Sure, we can fight such explosion by different ways of cheating, like high-level planning or using neural nets to predict most promising branches. But the main idea is that intelligence will eventually boil down to searching the best answer by trying - like evolution does.
How much work is "eventually" doing in that sentence, is my question. We already have machine learning systems in some fields (pharma, materials science) that greatly reduce the number of experiments researchers need to conduct to achieve a goal or get an answer. How low does the bound need to get?
I see a lot of discussion and speculation about "there's no way to get this right on the first try even for a superintelligence" but I don't think that's the right constraint unless you've already somehow contained the system in a way that only allows it a single shot to attempt something. In which case, you're most of the way to full containment anyway. Otherwise, the system may require additional trials/data/feedback, and will be able to get them, with many fewer such attempts than a human would need.
No one doubts that an ASI would have an easier time executing its plans than we could imagine but the popular claim is one-shot.
Yes, but I think it's important that when someone says, "Well I think one-shotting X is impossible at any level of intelligence," you can reply, "Maybe, but that doesn't really help solve the not-dying problem, which is the part that I care about."
I think the harder the theoretical doom plan it is the easier it is to control at least until alignment research catches up. It’s important because obsessing over unlikely scenarios that make the problem harder than it is can exclude potential solutions.
I do think it's plausible that e.g. nanotech requires some amount of trial-and-error or experimentation, even for a superintelligence. But such experimentation could be done quickly or cheaply.
But the main idea is that intelligence will eventually boil down to searching the best answer by trying - like evolution does.
Evolution is a pretty dumb optimization process; ordinary human level intelligence is more than enough to surpass its optimization power with OOM less trial and error.
For example, designing an internal combustion engine or a CPU requires solving some problems which might run into combinatorial explosions, if your strategy is to just try a bunch of different designs until you find one that works. But humans manage to design engines and CPUs and many other things that evolution couldn't do with billions of years of trial and error.
There might be some practical problems for which combinatorial explosion or computational hardness imposes a hard limit on the capabilities of intelligence. For example, I expect there are cryptographic algorithms that even a superintelligence won't be able to break.
But I doubt that such impossibilities translate into practical limits - what does it matter if a superintelligence can't crack the keys to your bitcoin wallet, if it can just directly disassemble you and your computer into their constituent atoms?
Maybe developing disassembling technology itself unavoidably requires solving some fundamentally intractable problem. But I think human success at various design problems is at least weak evidence that this isn't true. If you didn't know the answer in advance, and you had to guess whether it was possible to design a modern CPU without intractable amounts of trial and error, you might guess no.
Maybe developing disassembling technology itself unavoidably requires solving some fundamentally intractable problem
It’s very difficult to argue with most of the other claims if the base assumption is that this sort of technology is a) possible b) in one or few shots c) with reasonable for the planet compute.
1
1I've asked the mods to enable inline reacts for comments on this post.
If the feature is enabled (Now enabled!) You can react to each of the takes individually by highlighting them in the list below.
- Computational complexity theory does not say anything practical about the bounds on AI (or human) capabilities.
- From here on, capabilities research won't fizzle out (no more AI winters).
- Scaling laws and their implications, e.g. Chinchilla, are facts about particular architectures and training algorithms.
- "Human-level" intelligence is actually a pretty wide spectrum.
- Goal-direction and abstract reasoning (at ordinary human levels) are very useful for next-token prediction.
- LLMs are evidence that abstract reasoning ability emerges as a side effect of solving any sufficiently hard and general problem with enough effort.
- The difficulty required to think up algorithms needed to learn and do abstract reasoning is upper bounded by evolution.
- The amount of compute required to run and develop such algorithms can be approximated by comparing current AI systems to components of the brain.
- One of the purposes of studying some problems in decision theory, embedded agency, and agent foundations is to be able to recognize and avoid having the first AGI systems solve those problems.
- Even at human level, 99% honesty for AI isn't good enough.
5
4
2Claims like "ideal utility maximisation is computationally intractable"
This implies it could be impossible to know with given parameters and a function what the arg max is. I don’t see how you can get around this. I don’t understand what you are ultimately concluding is the result from that statement being false or not related to computation.
But that doesn't mean that any particular, non-parameterized instance of the problem cannot be solved some other way, e.g. by exploiting a regularity in the particular instance,
This is probably typically the equivalent of using a probabilistic solution or approximation. Exploiting a regularity means you are using the average solution for that regularity. Or that regularity has already been solved/is computable. I think the latter is somewhat unlikely.
using a heuristic or approximation or probabilistic solution,
This introduces accumulated uncertainty.
or that a human or AI can find a way of sidestepping the need to solve the problem entirely.
The whole world model is a chaotic incomputable system. Example: how people will react to your plan, given no certain knowledge of which people will observer your plan
that the main bottleneck on AI capabilities progress going forward will be researcher time to think up, design, implement, and run experiments.
It will be hardware, as it has always been. The current paradigm makes it seem like algorithms progress faster than hardware does and is bottlenecked by hardware. I don’t see any evidence against this.
Such an estimate, if accurate, would imply that the compute and energy requirements for human-level AGI are roughly approximated by scaling current AI systems by 5x.
Unsure about this claim. How are you relating neuron count to current systems? I think the actual bio anchors claim puts it at ~2040. 5x puts it within the next 2 years. We are also rapidly approaching the limits of silicon.
Evolution discovering something doesn’t necessarily mean it is easy. Just that it’s possible. We also have way more energy than is provably required for human level AGI.
An AI system that is only slightly superhuman might be capable of re-arranging most of the matter and energy in the visible universe arbitrarily
This claim seems fantastical. Ignoring the fact that we already have slightly superhuman narrow AI that don’t enable us to do any of that, and that groups of smart humans are slightly superhuman, the AI will have to reverse entropy and probably generate unlimited energy to do that.
A misaligned human level genius isn’t exactly scary either unless it’s directing us somehow and it gains a lot of influence. However history is full of these and yet we still stand.
LLMs are evidence that abstract reasoning ability emerges as a side effect of solving any sufficiently hard and general problem with enough effort.
Don’t see why this has to be the case. If language itself plays a role in development of abstract thought, then this could explain why LLMs in particular seem to have abstract thought.
Even at human level, 99% honesty for AI isn't good enough.
This claim isn’t particularly convincing. A lying politician lies 70% of the time. A righteous politician lies 30% of the time. I am also pretty confused by the story. What exactly is it portraying?
1
1
1
1I don’t understand what you are ultimately concluding is the result from that statement being false or not related to computation.
One thing I'm saying is that arguments of the form: "doing X is computationally intractable, therefore a superintelligence won't be able to do X" are typically using a loose, informal definition of "computationally intractable" which I suspect makes the arguments not go through. Usually because X is something like "build nanotech", but the actual thing that is provably NP-hard is something more like "solve (in full generality) some abstracted modelling problem which is claimed to be required to build nanotech".
Another thing is that even if X itself is computationally intractable, something epsilon different from X might not be. Non-rhetorical question: what does it matter if utility maximization is not "ideal"?
The whole world model is a chaotic incomputable system. Example: how people will react to your plan, given no certain knowledge of which people will observer your plan
This is a great example. non-computability has a precise technical meaning. If you want to link this meaning to a claim about a plan in the physical world or limit on ability to model human behavior, you have to actually do the work to make the link in a precise and valid way. (I've never seen anyone do this convincingly.)
Another example is the halting problem. It's provably undecidable, meaning there is no fully general algorithm that will tell you whether a program halts or not. And yet, for many important programs that I encounter in practice, I can tell at a glance whether they will halt or not, and prove it so. (In fact, under certain sampling / distribution assumptions, the halting problem is overwhelmingly likely to be solvable for a given randomly sampled program.)
One thing I'm saying is that arguments of the form: "doing X is computationally intractable, therefore a superintelligence won't be able to do X" are typically using a loose, informal definition of "computationally intractable" which I suspect makes the arguments not go through.
Not really, for an extreme example, simulating the entire universe is computationally intractable because it would require more resources than in the universe. By intractable I personally just mean it requires much more effort to simulate/guess than experiment. An obvious real world example is cracking a hash like you said. None of this has to do with NP hardness, it’s more like an estimate on the raw computational power required, and whether that exceeds the computational resources theoretically available. Another factor is also the amount of uncertainty the calculation will necessarily have.
Another thing is that even if X itself is computationally intractable, something epsilon different from X might not be.
Nothing that reduces to X can be computationally intractable or X isn’t. I don’t see any way around adding uncertainty by computing X-e which differs from X. Again this is just simply stated as approximations. In a model to build nano tech, an easy way this pops up is if there are multiple valid hypotheses given raw data, but you need high powered tools to experimentally determine which one is right based on some unknown unknowns. (Say a constant that hasn’t been measured yet to some requisite precision). This isn’t too contrived as it pops up in the real world a lot.
If you want to link this meaning to a claim about a plan in the physical world or limit on ability to model human behavior, you have to actually do the work to make the link in a precise and valid way.
Easy, lots of independent decision makers means there are an exponential number of states the entire system can be in. I’m sure there are also a lot of patterns in human behavior but humans are also diverse in their values and reactions that it would be difficult to know with certainty how some humans may react. Therefore you can only assign probabilities of what state an action might cause a system to transition to. A lot of these probabilities will be best guesses as well so there will be some sort of compounding error.
Concretely in a world domination plan, social engineering isn’t guaranteed to work. All it takes is for one of your actions to get the wrong reaction, i.e. evoke suspicion, and you get shut down. And it may be impossible to know for certain if you can avoid that ending.
I am not trying to relate it to computability theory, just showing that you get compounding uncertainty. Simulation of quantum dynamics of sufficient scale is an obvious example.
Another example is the halting problem. It's provably undecidable, meaning there is no fully general algorithm that will tell you whether a program halts or not. And yet, for many important programs that I encounter in practice, I can tell at a glance whether they will halt or not, and prove it so.
If you know whether a program falls in a set of of solvable programs, you’ve already solved the halting problem. I assume you still double check your logic. I also don’t see how you can prove anything for sufficiently large modules.
I don’t see why the halting problem is relevant here, nor do I see that paper proving anything about real world programs. It’s talking about arbitrary tapes and programs. I don’t see how it directly relates to real life programming or other problem classes.
And if you want to talk about NP hardness, it seems that decision makers regularly encounter NP hard problems. It’s not obvious that they’re avoidable or why they would be. I don’t see why an ASI would for example be able to avoid the hardness of determining optimal resource allocation. The only side step I can think of is not needing resources or using a suboptimal solution.
This paper seems to explain it better than I can hope to: https://royalsocietypublishing.org/doi/10.1098/rstb.2018.0138
Non-rhetorical question: what does it matter if utility maximization is not "ideal"?
Getting stuck in a local maxima. This seems to undermine the whole atoms line of thought.
How are you relating neuron count to current systems?
I'm not; the main point of the comparison method I propose is that it sidesteps the need to relate neurons in the brain to operations in AI systems.
The relevant question is what fraction of the brain is used to carry out a high-level task like speech recognition or visual processing. How to accurately measure that fraction might involve considering the number of neurons, energy consumption, or synaptic operations in a particular region of the brain, as a percent of total brain capacity. But the comparison between brains and AI systems is only on overall performance characteristics at high-level tasks.
5x puts it within the next 2 years.
I haven't actually done the research to estimate what fractions of the brain are used to do which tasks; 5x was just an example. But it wouldn't surprise me if there is enough existing compute already lying around for AGI, given the right algorithms. (And further, that those algorithms are not too hard for current human researchers to discover, for reasons covered in this post and elsewhere.)
I'm not; the main point of the comparison method I propose is that it sidesteps the need to relate neurons in the brain to operations in AI systems.
At best it may show we need a constant factor amount less than the brain has (I am highly doubtful about this claim) to reach its intelligence.
And no one disputes that we can get better than human performance at narrower tasks with worse than human compute. However, such narrow AI also augment human capabilities.
How exactly are you side stepping computation requirements? The brain is fairly efficient at what it has to do. I would be surprised if given the brains constraints, you could get more than 1 OOM more efficient. A brain also has much longer to learn.
But it wouldn't surprise me if there is enough existing compute already lying around for AGI, given the right algorithms. (And further, that those algorithms are not too hard for current human researchers to discover, for reasons covered in this post and elsewhere.)
Do you have any evidence for these claims? I don’t think your evolution argument is strong in proving that they’re easy to find. I am also not convinced that current hardware is enough. The brain is also far more efficient and parallel at approximate calculations than our current hardware. The exponential growth we’ve seen in model performance has always been accompanied by an exponential growth in hardware. The algorithms used are typically really simple which makes them scalable.
Maybe an algorithm can make the computer I’m using super intelligent but I highly doubt that.
Also I think it would be helpful to retract the numbers or at least say it’s just a guess.
1LLMs are evidence that abstract reasoning ability emerges as a side effect of solving any sufficiently hard and general problem with enough effort.
I'd probably not claim this, or at least significantly limit the claim here, because of some new results on Transformers/LLMs, and apparently they don't actually do multi-step reasoning, but instead develop short cuts, and importantly can't implement recursive algorithms, which is really important.
Tweets below:
This is actually a fairly important limitation of LLMs, and appears to sort of vindicate the LLM skeptics like Yann Lecun and Gary Marcus, in that LLMs don't actually reason on multi-step problems all that well.
[This comment is no longer endorsed by its author]
It seems like it's easy to break this limitation by writing prompts that break a problem into pieces, then calling a new instance of the LLM to solve each piece and then to provide the answer given the step by step reasoning from previous prompts. The SmartGPT does something like this, and achieves vastly better performance on the logical reasoning benchmarks it's been tested on.