This is my offering to the Future Fund's "A.I. Worldview Prize" - and I stand as an outsider who is unlikely to move the needle; I hope this angle of approach is noticed there, at least.
Our Last Limitations?
The AGI with feelings and motives is irrelevant to my concerns; I worry about a machine that can add to its skill-set without bounds. The key functional constraints on this are 'catastrophic forgetting' and the difficulty extracting causal relationships & formulae from observations. Once those twin pillars fall, a single AGI can successively add-to its competencies, while finding symbolic parallels which further compress and unify its understanding. Even such a 'Vulcan' AGI could be a mis-aligned monster.
I don't see those 'forgetting & formulating' problems taking very long to solve; my personal timeline is 5-10yrs. with greater than half likelihood (bad stuff can happen, but this industry is not about to winter). After those hurdles, the issue of 'getting to brain-scale' will be found irrelevant - because human brains are bloatware. To explain:
We worry that, because each neuron in our own brains has complexity that takes ~200 artificial neurons to simulate, and because we have 100 trillion synapses or so, then 'getting to human brain performance' will also require getting to some 20 million billion weights and biases. Yet, all of the evidence so far is a gauntlet challenging this concern. Consider that language models are able to out-perform the average human (we are an AGI) while containing some 1,000 times fewer connections than us. Even driverless vehicles have a better safety-record than the average human, and last I checked, those were not running an AGI with trillions of parameters.
Realistically, we should expect the human brain to be hyper-redundant - look at our plasticity! And the rate of loss, damage, over the course of our reality-bonked lives! And, our brains learn from startlingly few examples, which I suspect is related to its immense over-parameterization. (This has an impact on AGI policy, too - We won't want AGI zero-shotting situations that we ourselves haven't grasped yet; "over-parameterizing an AGI with sparse training data to zero-shot the unexpected" is the definition of insane risk!)
So, near-term compute capacity will be plenty to run super-human AGI... and it'll be crap.
The Narrow Revolution
Consider that an AGI will need more time training compared to a narrow intelligence, with more tedious checks upon its behaviors, considering the myriad tasks it can be asked to perform, and all of this will take longer to run-through its enormous network, leading to longer lag before it answers, while requiring thousands of times more compute. Here is my bet:
"If we have equal training data and compute, memory, etc. - and we have a real-world business or military objective, then my 1,000 distinct instances of differentiated narrow networks will outperform your bloated, slow, single instance of an AGI. If you pick AGI, you lose."
General intelligence was only called-upon once, during a period of repeated ice ages that gave advantage to the fastest adaptations, while narrow intelligence is preferred across the natural world. Industry is showing the same persistent pattern; Narrow wins.
With narrow AI, there are still immense risks; yet, we can ensure safety from 'take-over' by restricting ourselves to non-temporal intelligence. As per standard practice already, a network's weights are frozen once it is deployed, which ensures that it cannot form new memories. A frozen network cannot execute a multi-stage plan, because it doesn't know which steps it already completed. It can only respond to each prompt; it cannot form a strategy to keep you from turning it off. No Robo-Overlords.
That is my second bet: "We will realize that prompted, task-focused narrow intelligences do almost everything we feel comfortable with them doing, while AGI doesn't provide any performance benefits and it exposes us to known catastrophic risk. Therefore, we never bother with AGI (after proving that we can do it). The real safety issues are from poor alignment and mis-use."