Well, it's an interesting opinion piece. But I have no idea what reference to quantum computing and the Landauer limit is supposed to prove. Intelligence explosion is in part about compute overhang, sure, but we can have compute overhang without quantum computing, just by being inefficient about training.
Basically, it means that mildly superhuman intelligences trained by the Chinchilla scaling laws must use physical actions like trying to gain more energy for more compute, and it can't just try to get more intelligent by orders of magnitude in minutes, hours or weeks. Such takeovers will take years to gain a significant fraction of the global economy and power
In other words, Landauer's Limit acts as a firebreak for fast-takeoff, because it just can't get energy from nowhere, and physical systems do not scale fast enough for the Intelligence Explosion to work.
AI will takeover, it's just not as fast as Yudkowsky thinks it will.
Landauer's limit only 'proves' that when you stack it on a pile of assumptions a mile high about how everything works, all of which are more questionable than it. It is about as reliable a proof as saying 'random task X is NP-hard, therefore, no x-risk from AI'; to paraphrase Russell, arguments from complexity or Landauer have all the advantages of theft over honest toil...
Well, the assumptions are that everyone is using classical computers, thermodynamics exists, thus they must get energy from somewhere (albeit with superhuman skill). That's the list of assumptions necessary to prove the intelligence Explosion scenario wrong.
I've already accepted that a superhumanly intelligent AI can take over the world in about several years, which is arguably somewhat too fast for the public to realize, and AI X-risk chances are in my own view oscillating towards 30-60% this century, so this is still no joking matter. AI Alignment matters still, it still needs to get more researchers and money.
Now I will be wrong if quantum/reversible computers become more practical than they are currently, and companies can reliably train AI on quantum/reversible computers.
My post is more of a retrospective, as well as saying that things take time to manifest it's full impact.
EDIT: My point is, the Singularity as envisioned by Ray Kurzweil and John Smart has already happened, it just takes time from now to the end of the century. The end of the century will be very weird. It's just more continuously weird, with a discontinuity unlocking new possibilities that are then claimed by continuous effort by AGI and ASI.
That's the list of assumptions necessary to prove the intelligence Explosion scenario wrong.
No, it's not. As I said, a skyscraper of assumptions each more dubious than the last. The entire line of reasoning from fundamental physics is useless because all you get is vacuous bounds like 'if a kg of mass can do 5.4e50 quantum operations per second and the earth is 6e24 kg then that bounds available operations at 3e65 operations per second' - which is completely useless because why would you constrain it to just the earth? (Not even going to bother trying to find a classical number to use as an example - they are all, to put it technically, 'very big'.) Why are the numbers spat out by appeal to fundamental limits of reversible computation, such as but far from limited to, 3e75 ops/s, not enough to do pretty much anything compared to the status quo of systems topping out at ~1.1 exaflops or 1.1e18, 57 orders of magnitude below that one random guess? Why shouldn't we say "there's plenty of room at the top"? Even if there wasn't and you could 'only' go another 20 orders of magnitude, so what? what, exactly, would it be unable to do that it would if you subtracted or added 10 orders of magnitude* and how do you know that? why would this not decisively change economics, technology, politics, recursive AI scaling research, and everything else? if you argue that this means it can't do something in seconds and would instead take hours, how is that not an 'intelligence explosion' in the Vingean sense of being an asymptote and happening far faster than prior human transitions taking millennia or centuries, and being a singularity past which humans cannot see nor plan? Is it not an intelligence explosion but an 'intelligence gust of warm wind' if it takes a week instead of a day? Should we talk about the intelligence sirocco instead? This is why I say the most reliable part of your 'proof' are also the least important, which is the opposite of what you need, and serves only to dazzle and 'Eulerize' the innumerate.
* btw I lied; that multiplies to 3e75, not 3e65. Did you notice?
I was talking about irreversible classical computers, where Landauer bounds it much much harshly, not quantum computers relying on the much looser Margolus-Levitin limit.
To put this in perspective, there's a 35 order of magnitude or so difference between what a quantum computer's limit is and what a classical computer's limit is.
Here's a link:
https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know
While the most pessimistic conclusions are probably wrong (I think they will accept the energy cost of 300 watts-5 kilowatts to increase computation by one or 2 orders of magnitude, since intelligence is very valuable and such a thing would lead to a Vernor Vinge-like singularity) this is a nice post for describing the difference.
So it actually supports my post here, since I talked about how quantum/reversible computers would favor the intelligence Explosion story. I am pessimistic on quantum/reversible computers being practical before 2100, which is why the accelerating change story is favored.
So you and me actually agree here, so are we getting confused about something.
And black hole computers are possible, but realistically will be developed post-singularity.
I think from an x-risk perspective the relevant threshold is: when AI no longer needs human researchers to improve itself. Currently there is no (publicly known) model which can improve itself fully automatically. The question we need to ask is, when will this thing get out of our control? Today, it still needs us.
This seems to assume that those scaling laws are a hard bound, rather than just a contingent fact about a particular machine learning method.
This assumes a fixed scaling law. One possible way of improving oneself could be to design a better architecture with a better scaling exponent.
Considering that there is, in fact, no singularity at this point, you have clearly jumped the gun on this. There is no proof that actually feasible scaling will get us even a notable fraction of the way to singularity, so you can hardly proclaim that no singularity has been shown to be wrong, or which pro-singularity school is right. Your reasons for believing as you do aren't even really in the essay, so it's a castle built without any foundation.
Not having good evidence is no reason at all to wilfully ignore bad-in-some-respect evidence. It's just usually not worth bothering with. Not always, and the distinction needs to be drawn the hard way.
Doesn't apply when you are trying to justify something incredibly unrelated thing...especially when you are clearly wrong, and clearly claim that something has happened. Singularity is after so many steps and illogical leaps from where we are that there is little real evidence (all of which points away from imminent singularity.) Your claim is a completely untrue basis for the entire thing. Make a real argument, don't simply beg the question.
I'm not the OP, I'm not making claims about technological singularity in this thread. I'm making claims about the policy of dismissing unfounded claims of any kind.
I admit, I didn't actually check the name of the original poster versus yours. It's not actually relevant to anything but my use of the pronoun 'you', and even that works perfectly well as a general 'you' (which is a quirk of English that is a bit strange). The rest of my response stands, it is a terrible argument to make about evidence. The original post is a textbook example of begging the question. Confident claims should not be made on effectively non-existent evidence.
it is a terrible argument to make about evidence
What do you mean by "argument about evidence"? I'm not arguing that the evidence is good. I'm arguing that in general response to apparently terrible evidence should be a bit nuanced, there are exceptions to the rule of throwing it out, and some of them are difficult to notice. I'm not even claiming that this is one of those exceptions. So I'd call it an argument about handling of evidence rather than an argument about evidence.
What's wrong with the argument about handling of evidence I cited?
You are definitely making an argument about the evidence's status. What's wrong is that you are smuggling in the assumption that there is evidence that is simply flawed rather than a lack of evidence. To 'willfully ignore' evidence requires that there be real evidence! The link appears to be a non-sequitur otherwise. It is hardly an isolated demand for rigor to state that there must be evidence at all on the matter being something that is clearly happening to have a retrospective on 'why it HAPPENED, who WAS wrong, and who WAS right'. Thus, your argument is terrible.
There is a useful distinction between an argument not applying, that is being irrelevant, flawed-in-context, and it being terrible, that is invalid, flawed-in-itself.
you are smuggling in the assumption that there is evidence that is simply flawed rather than a lack of evidence
This is not an assumption that would benefit from being sneakily smuggled in. Instead this is the main claim I'm making, that even apparently missing evidence should still be considered a form of evidence, merely of a very disagreeable kind, probably not useful, yet occasionally holding a germ of truth, so not worth unconditionally dismissing.
The post I cited is targeted at impressions like this, that something is not a legitimate example of evidence, not worthy of consideration. Perhaps it's not there at all! But something even being promoted to attention, getting noticed, is itself a sort of evidence. It may be familiar to not consider that a legitimate form of evidence, but it's more accurate to say that it's probably very weak (at least in that framing, without further development), not that it should never, on general principle, interact with state of knowledge in any way (including after more consideration).
Incidentally, evidence in this context is not something that should make one believe a claim with greater strength, it could as easily make one believe the claim less (even when the piece of evidence "says" to believe it more), for that is the only reason it can be convincing in the first place. So it's not something to defend skepticism from. It just potentially shifts beliefs, or sets the stage for that eventually happening, in a direction that can't be known in advance, so its effect couldn't really be read off the label.
You did not make that claim itself. The link is not even on target for what you are claiming. It is a rant against unachievable demands for rigor, not a claim that all demands for basic coherency are wrong; the clear reason, which I already stated for why it isn't evidence is because it is based entirely on a clearly false premise. If it doesn't apply to the question at hand, it isn't evidence! This is basic reasoning. When a thing hasn't happened, and there is no evidence it is imminent, we can't possibly have an actual retrospective, which is what this claims to be. If this were simply a post claiming that they think these schools are likely to be right, and these wrong, for some actual reasons, that would be a completely different kind of thing than this, which is entirely based on a clearly false claim of fact.
which is entirely based on a clearly false claim of fact
The failure mode is (for example) systematic inability to understand a sufficiently alien worldview that keeps relying on false claims of fact. The problem is attention being trained to avoid things that are branded with falsity, bias, and fallacy, even by association. One operationalization of a solution is Ideological Turing Test, diligently training to perchance eventually succeed in pretending to be the evil peddler of bunkum. Not very appealing.
I prefer the less specifically demanding method that is charity. It's more about compartmentalizing the bunkum without snuffing it out. And it has less hazardous applications.
This is useless until it isn't, and then it's crucial.
As a clarification of my own worldview on AI and the Singularity, I am basically saying that we are in the early part of the exponential curve, and while AI's short term effects are overhyped in say 5 years, in 50-100 years AI will change the world so much that it can be called a Singularity.
My biggest evidence comes from the WinoGrande dataset, which achieves 70.3% by GPT-3 without fine-tuning. While BERT went back to random chance-like accuracy, GPT has some common sense (Though worse than human common sense).
Also, GPT-3 can meta-learn languages the first time it receives the data.
And yeah, Vladimir Nesov correctly called my feels on the no fire alarm issue for AI. The problem of exponentials is that in Nesov's words,
This is useless until it isn't, and then it's crucial.
And that's my theory of why AI progress looks so slow: We're in the 70s era for AI, and the people using AI are companies.
There is an opposition between skepticism and charity. Charity is not blanket promotion of sloppy reasoning, it's an insurance policy against misguided stringency in familiar modes of reasoning (exemplified by skepticism). This insurance is costly, it must pay its dues in attention to things that own reasoning finds sloppy or outright nonsense, where it doesn't agree that its stringency would be misguided. This should still be only a minor portion of the budget.
It contains the damage with compartmentalization, but reserves the option of accepting elements of alien worldviews if they ever grow up, which they sometimes unexpectedly do. The obvious danger in the technique is that you start accepting wrong things because you permitted yourself to think them first. This can be prevented with blanket skepticism (or zealous faith, as the case may be), hence the opposition.
(You are being very sloppy in the arguments. I agree with most of what deepthoughtlife said on object level things, and gwern also made relevant points. The general technique of being skeptical of your own skepticism doesn't say that skepticism is wrong in any specific case where it should be opposed. Sometimes it's wrong, but it doesn't follow in general, that's why you compartmentalize the skepticism about skepticism, to prevent it from continually damaging your skepticism.)
I'm honestly not sure you'll get anything out of my further replies since you seem to have little interest in my actual points.
My original comment on the blatant falsehood that was the premise original post was as charitable as reasonably possible considering how untrue the premise was. If there was any actual evidence supporting the claim, it could have been added by an interlocutor.
I simply pointed out its most glaring flaws in plain language without value judgment. The main premise was false for even the most charitable version of the post. To be any more charitable would have required me to rewrite the entire thing since the false premise was written so deeply into the fabric of things. If they had restricted themselves to plausible things, it is quite possible it would have been a useful post, but this wasn't.
I neither have the time nor the inclination to write out the entirety of what the argument should have been. I don't really believe in Ideological Turing Tests, just like I don't believe that Turing tests are a great measure for AI. It's not that there aren't uses for it, it's just that those are niche (though an AI that reliably passes the Turing Test could make a lot of money for its creators.). I don't have forever to fix bad arguments.
A basic outline of the initial argument in the original post is:
1)The singularity is moving slowly, but already upon us. We are in takeoff.
2)The takeoff will remain slow (though quick enough to be startling.)
3)Thus no singularity is clearly wrong.
4)Fast take off is also clearly false.
5)Scaling is all that matters.
6)Since scaling will be so expensive due to Landauer's principle, high end AGI will happen, but not in private hands for a long time.
There are several more 'implications' of this that I won't bother writing because they clearly rely on the former things.
1 is meant to prove 3,4,5 directly, while 2 absolutely needs 1 to be true.
2 and 5 are meant to set the stage for 6..
These were largely bare assertions (which there is a place for). I objected that point 1 was clearly false, rendering points 3,4,5,6 impossible to tell based on this argument structure and the available evidence, and 2 clearly meaningless (since it is defined falsely, something cannot remain a certain way if it is not already that way). (Even though I agree with 4 and 6, and and I could rewrite their argument to make 2 much more sensible.) Since there was no extra evidence beyond a known unsound argument, that rendered the rest of it irrelevant. The leap from 1 to 5 would be quite weak even if not for the fact the 1 is false.
7)And thus we should be very scared of not having any wake up call for when AI will become very dangerous.
7 is clearly unsupported at this point since all of the assumptions leading here are useless.
I like a good hypothetical, but I don't really have any interest in continuing to engage with things that are that wrong factually, and won't admit it.
'The moon really is made of cheese, so what does that mean for how we should approach the sun?' That is literally the level of uselessness I find this approach to detailing the state of AI and how that relates how to approach alignment to have. (Like I said, the version in my original comment was the charitable one.)
I could make an argument for or against anything they claim in this post, but it wouldn't be a response to what they actually wrote, and I don't see how that would be useful.
Cute terminology I came up with while talking about this recently: 'zoom-foom gap'
I like to call accelerating change period where AI helps accelerate further AI but only via working with humans at a less-than-human-contribution level the 'zoom' in contrast to the super-exponential artificial-intelligence-independently-improving-itself 'foom'. Thus, the period we are currently is the 'zoom' period, and the oh-shit-we're-screwed-if-we-don't-have-AI-alignment period is the foom period. When I talk about the future critical juncture wherein we realize could initiate a foom at then-present technology, but we restrain ourselves because we know we haven't yet nailed AI alignment, the 'zoom-foom gap'. This gap could be as little as seconds while a usually-overconfident capabilities engineer pauses just for a moment with their finger over the enter key, or it could be as long as a couple of years while the new model repeatedly fails the safety evaluations in its secure box despite repeated attempts to align it and thus wisely doesn't get released. 'Extending the zoom-foom gap' is thus a key point of my argument for why we should build a model-architecture-agnostic secure evaluation box.
TLDR: I like having a reason to use the term 'zoom-foom gap'.
TL;DR of this post: Accelerating change and Event Horizon were the most accurate schools, with Intelligence Explosion proving to be interestingly wrong (Discontinuities only make a new field for AI get off the ground, not solve the entire problem ala Nuclear weapons, and scaling does show discontinuities, but only in the sense that an intractable problem or paradigm becomes possible, not chaining discontinuities to entirely solve the problem at a superhuman level.) The non-singularitian scenarios were wrong in retrospect, but in the 2000s, it would have been somewhat reasonable to say that no singularity was going to happen.
In other words, the AI-PONR has already happened and we are living in a slow rolling singularity already.
Long answer: That's the topic of this post.
Back in 2007, before deep learning and AI actually solved real problems and the AI winter was going strong, Eliezer Yudkowsky over at www.yudkowsky.net placed the Singularitians into 3 camps, which I will reproduce here for comparison:
Core claim: Our intuitions about change are linear; we expect roughly as much change as has occurred in the past over our own lifetimes. But technological change feeds on itself, and therefore accelerates. Change today is faster than it was 500 years ago, which in turn is faster than it was 5000 years ago. Our recent past is not a reliable guide to how much change we should expect in the future.
Strong claim: Technological change follows smooth curves, typically exponential. Therefore we can predict with fair precision when new technologies will arrive, and when they will cross key thresholds, like the creation of Artificial Intelligence.
Advocates: Ray Kurzweil, Alvin Toffler(?), John Smart
Core claim: For the last hundred thousand years, humans have been the smartest intelligences on the planet. All our social and technological progress was produced by human brains. Shortly, technology will advance to the point of improving on human intelligence (brain-computer interfaces, Artificial Intelligence). This will create a future that is weirder by far than most science fiction, a difference-in-kind that goes beyond amazing shiny gadgets.
Strong claim: To know what a superhuman intelligence would do, you would have to be at least that smart yourself. To know where Deep Blue would play in a chess game, you must play at Deep Blue’s level. Thus the future after the creation of smarter-than-human intelligence is absolutely unpredictable.
Advocates: Vernor Vinge
Core claim: Intelligence has always been the source of technology. If technology can significantly improve on human intelligence – create minds smarter than the smartest existing humans – then this closes the loop and creates a positive feedback cycle. What would humans with brain-computer interfaces do with their augmented intelligence? One good bet is that they’d design the next generation of brain-computer interfaces. Intelligence enhancement is a classic tipping point; the smarter you get, the more intelligence you can apply to making yourself even smarter.
Strong claim: This positive feedback cycle goes FOOM, like a chain of nuclear fissions gone critical – each intelligence improvement triggering an average of>1.000 further improvements of similar magnitude – though not necessarily on a smooth exponential pathway. Technological progress drops into the characteristic timescale of transistors (or super-transistors) rather than human neurons. The ascent rapidly surges upward and creates superintelligence (minds orders of magnitude more powerful than human) before it hits physical limits.
Advocates: I. J. Good, Eliezer Yudkowsky
There's a link to it to verify my claims on this topic: https://www.yudkowsky.net/singularity/schools
But we need a control group, given that these schools are biased toward believing a great change or rapture is coming, even in it's most mild forms. So I will construct a nonsingularitian school to serve as a control group. There will be positive and negative scenarios here.
Core Claim: The world won't mind upload into digital life or have an AI God, nor will a catastrophe happen soon. It's instead the boring future predicted by the market, which means closer and closer to linear or even sub-linear growth. The end of even a smooth exponential, combined with drastically dropping birthrates everywhere, means permanent stagnation. At this juncture in 2050, 2 things can happen: Slow extinction over thousands of years and continual failure to maintain birth rates, or the world stabilizes permanently into a 2.1 birthrate, never increasing it's economy or technology again but still overall a rich world, meaning they are still satisfied with this state of affairs until the Sun burns out.
Strong Claim: Humans are still human, and human nature has been only weakly suppressed, and stagnation similar to the ancient era has returned, but they don't care, since the population has also stagnated or slowly declining.
Advocates: The Market as a whole mostly expects the no singularity positive scenario.
Core claim: The 21st century turns out to be a disappointment in all respects. We do not merge with the Machine God, nor do we descend back into the Olduvai Gorge by way of the Fury Road. Instead, we get to experience the true torture of seeing the conventional, mainstream forecasts of all the boring, besuited economists, businessmen, and sundry beigeocrats pan out.
Strong Claim: Human genetic editing is banned by government edict around the world, to “protect human dignity” in the religious countries and “prevent inequality” in the religiously progressive ones. The 1% predictably flout these regulations at will, improving their progeny while keeping the rest of the human biomass down where they believe it belongs, but the elites do not have the demographic weight to compensate for plummeting average IQs as dysgenics decisively overtakes the Flynn Effect.
We discover that Kurzweil’s cake is a lie. Moore’s Law stalls, and the current buzz over deep learning turns into a permanent AI winter. Robin Hanson dies a disappointed man, though not before cryogenically freezing himself in the hope that he would be revived as an em. But Alcor goes bankrupt in 2145, and when it is discovered that somebody had embezzled the funds set aside for just such a contingency, nobody can be found to pay to keep those weird ice mummies around. They are perfunctorily tossed into a ditch, and whatever vestigial consciousness their frozen husks might have still possessed seeps and dissolves into the dirt along with their thawing lifeblood. A supermall is build on their bones around what is now an extremely crowded location in the Phoenix megapolis.
For the old concerns about graying populations and pensions are now ancient history. Because fertility preferences, like all aspects of personality, are heritable – and thus ultracompetitive in a world where the old Malthusian constraints have been relaxed – the “breeders” have long overtaken the “rearers” as a percentage of the population, and humanity is now in the midst of an epochal baby boom that will last centuries. Just as the human population rose tenfold from 1 billion in 1800 to 10 billion by 2100, so it will rise by yet another order of magnitude in the next two or three centuries. But this demographic expansion is highly dysgenic, so global average IQ falls by a standard deviation and technology stagnates. Sometime towards the middle of the millenium, the population will approach 100 billion souls and will soar past the carrying capacity of the global industrial economy.
Then things will get pretty awful.
But as they say, every problem contains the seed of its own solution. Gnon sets to winnowing the population, culling the sickly, the stupid, and the spendthrift. As the neoreactionary philosopher Nick Land notes, waxing Lovecraftian, “There is no machinery extant, or even rigorously imaginable, that can sustain a single iota of attained value outside the forges of Hell.”
In the harsh new world of Malthusian industrialism, Idiocracy starts giving way to A Farewell to Alms, the eugenic fertility patterns that undergirded IQ gains in Early Modern Britain and paved the way to the industrial revolution. A few more centuries of the most intelligent and hard-working having more surviving grandchildren, and we will be back to where we are now today, capable of having a second stab at solving the intelligence problem but able to draw from a vastly bigger population for the task.
Assuming that a Tyranid hive fleet hadn’t gobbled up Terra in the intervening millennium.
Advocates: Anatoly Karlin
Age of Malthusian Industrialism series:
http://www.unz.com/akarlin/short-history-of-3rd-millennium/
http://www.unz.com/akarlin/where-do-babies-come-from/
http://www.unz.com/akarlin/breeders-revenge/
http://www.unz.com/akarlin/breeding-breeders/
http://www.unz.com/akarlin/the-geopolitics-of-the-age-of-malthusian-industrialism/
http://www.unz.com/akarlin/world-population/
Now that we finished listing out the scenarios, we can ask the question, who was right from 15-20 years ago now that things happened?
The non-singularitians were ultimately wrong, but we shouldn't be too harsh due to hindsight biasing our estimates. That said, the development of Deep Learning managing to beat professional humans at Go was significant because it got investment by capitalists that gave billions of dollars to AI that infused things at the right time, and that money is usually far more stable than government money, essentially crushing AI winter issues once and for all. This is also a fairly continuous rather than discontinuous story.
But beyond stable money, why was it the right time? The answer is the bitter lesson by Richard Sutton, and basically that means compute, more than algorithms or instincts, matters for intelligence. And we finally have enough compute to actually simulate intelligence. Combine this with real money from capitalists, and things would explode exponentially fast. There's another post which at least shows why we have failed routinely to get AGI with classical computers, but not now: We now can get very close to the Landauer limit, with perhaps an OOM more than the brain for 300 Watts on a personal computer, when all is said and done about the absolute limit. Here's the link:
https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know
While the most pessimistic conclusions are challenged pretty well in the comments to the point that I think they aren't going to be too much of a barrier in practice, it is correct that the limits do essentially mean that slow takeoff or the Accelerating Change story was the correct model of growth in AI.
GOFAI failed because it required quantum computers to come pretty fast, since they are the only computers known to be reversible, circumventing Landauer's Limit and instead operating on the Margolus-Levitin Limit, which is far more favorable to the Intelligence Explosion story. Unfortunately, Quantum computing is coming much slower than necessary to support the Intelligence Explosion thesis, so it now has mostly no longer worked.
But mostly didn't work as a story didn't mean it didn't entirely work, and there does seem to be a threshold type effect that Intelligence Explosion predicts would happen, and as models scale in compute, they get from not being able to do something at all to doing something passably well. That is a threshold discontinuity, but crucially instead of being the entire process of intelligence amplification like I. J. Good and Eliezer Yudkowsky thought, it only lets AI make something possible that used to be unatomatable by AI be automatable, similar to how human language suddenly appeared in the Cognitive Revolution, but didn't guarantee that they would dominate the world 69,800 years later in the Industrial Revolution, which was the start of a slow takeoff of humanity where humans over 2 centuries gradually seperate more and more from nature, culminating in the early 2010s AI-PONR gradually taking away what control nature and evolution has left.
https://www.lesswrong.com/users/daniel-kokotajlo uses PONR as the time period when AI risk-reducing efforts essentially become much less valuable due to AI Takeover being inevitable. Daniel Kokotajlo places the PONR in the 2020s, while I place it in the mid-2010s-Early 2020s. My PONR estimate is basically from the time that Go was demolished by AI to the Chinchilla scaling paper. That is my own PONR because once capitalism invests billions of dollars into something mature like AI, it's a good predictor that it's in fact important enough such that this will be adopted in time. It also addresses the AI winter issues because now there's a stable source of funding for AGI that isn't government fully funding AI research.
A final ode to the Event Horizon story. Event Horizon does appear to be correct in that even mild scaling like 3x chimp brain creates effectively an event horizon scenario, where chimps can't even understand what a human can do, let alone try to replicate it in all but the most primitive details, and this gets worse once we exclude the most intelligent animals like primates, whales and some other groups of animals, where a worm doesn't even begin to understand anything at all about the human, or even an orca or komodo dragon, far easier to understand, and yet they can't understand. And this is why AI risk is so bad, because we will never understand what an AI can do, because the gap in intelligence is essentially more like the human-animal difference than the internal differences in human intelligence, which at least are bounded to at most 0.25x-1.9x the average. And when a more powerful group encounters a less powerful group, the default outcome is disastrous for the less powerful group. It's not the default for a more powerful group not to treat the less powerful group badly instead of well.
The biggest positive implication is that an AI research will take time to produce superintelligence, so we do have some time. We can't waste that time, given the near-inevitability of AGI and superintelligence, but it does mean that by and large, the intelligence revolution will spin up slowly at the start, so we do have a chance to influence it. So AI Alignment needs more researchers and more funding right now.
Next, up to a point, it is far safer to do empirical research than theorists like MIRI like to think. In fact, we will need short feedback loops in order to make the best uses out of our time, and we need productive mistakes to get at least a semblance of a solution. Thus this post:
Is entirely the wrong way to go about things in an Accelerating Change world, where there isn't a lot of sharp left turn discontinuities, and the ones that can be gotten won't solve the entire problem either for AI, so that post is irrelevant.
The next stage of Alignment work is a world where MIRI transforms itself purely into an awareness center for AI risk, not as a research organization unto themselves anymore. The next stage of work looks a lot more like Ajeya Cotra's sandwiching and empirical work, not Eliezer Yudkowsky's MIRI team's theoretical work.
One slight positive implication: At best, due to the Landauer limit, the general population will only have AGIs, and superintelligence will remain outside of individual hands for a long time to come. Thus, the scenario where a rogue person gets superintelligence in their basement is entirely impossible.
One negative implication is that the No Fire Alarm scenario is essentially the default condition. That's because as Ray Kurzweil saw correctly, people's models for change are linear when it's actually exponential. Combine this with the fact that we are on the early slope of such a curve, and you can't tell whether it will be a big deal or a small deal without serious thought. So the No Fire Alarm condition will continue until it's too late. Here's a link:
https://intelligence.org/2017/10/13/fire-alarm/
One final negative implication: Our brains don't handle x-risk/doomsday very well, or it's inverse near-utopia and truth be told, there are no good answers for your mental health, because the situation your brain is in is so-far off-distribution ala Extremal Goodhart that an actual, tangible scenario of doom or near-utopia is not one it has been designed to handle very well. I unfortunately don't have good answers here.