If you now got convinced that survival and future of everyone you know, including your own, depends on your immediate actions after reading this; you would not hesitate to put your daily tasks on halt to study and meditate on this text as if it was a live message being...
Thank you. I set to write something clear and easy to read that could serve as a good cornerstone to decisive actions later on and I still think I accomplished that fairly well.
The paper starts with the assumption that humans will create many AI-agents and assign some of them selfish goals and that combined with competitive pressure and other factors may presumably create a Molochy -situation where most selfish and immoral AI's will propagate and evolve - leading to loss of control and downfall of the human race. The paper in fact does not advocate the idea of a single AI foom. While the paper itself makes some valid points it does not answer my initial question and critique of OP.
But WHY would the AGI "want" anything at all unless humans gave it a goal(/s)? If it's a complex LLM-predictor what could it want besides calculate a prediction of its own predictions? Why by default it would want anything at all unless we assigned that as a goal and turned it into an agent? IF AGI got hell bent on own survival and improvement of itself to maximize goal "X" even then it might value the informational formations of our atoms more than the energy it could gain from those atoms, depending on what "X" is. Same goes for other species: evolution itself holds information. Even in case of a rogue AGI for at least some time window we could have something to offer.
A sufficiently capable AI takes you apart instead of trading with you at the point that it can rearrange your atoms into an even better trading partner.[1] And humans are probably not the optimal trading partners.
Probably? Based on what?
Humans are slow and petty creatures evolved to argue, collect stuff, hold tools and run around. We are not built to process raw information. Internet, as remarkable as it is - is mostly an echo chamber where people usually seek confirmation and reassurance rather than exploring frontiers of new modes of existing. Go on any forum and you will notice the same questions and ideas being expressed regularly regardless if there's a FAQ explaining everything. On less frequent intervals someone rediscovers that which countless others have rediscovered before them, but without knowing it seems like some mysterious and novel path of reason. This too has been said and written elsewhere so am... (read more)
I got your notes confused with the actual publisher's notes and it made sense when I figured that you took and combined the quotes from their site. I also analyzed the data. "At least 10%" should actually be "10%". The questionnaire was a free form. I think it's not fair to pair "738 responses, some partial, for a 17% response rate" with these concrete questions. 149 gave an estimate to the first question and 162 to the second question about the extinction. 62 people out of 162 assigned the second question 20% or higher probability.
Where did you get the "more than 10%" part from?
Right, it says the results summary that "48% of respondents gave at least 10% chance of an extremely bad outcome".
The survey had two questions. I took a deeper dive into it. The questions were as following:
1) "What probability do you put on future AI advances causing human extinction or similarly
permanent and severe disempowerment of the human species? "
Of those who answered the median probability was 5%. Summary says:
... (read more)The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. This is the same as it was in 2016 (though Zhang et
I've added your comment in full and another strike-through to change "a median 10% chance" into "at least a 10% chance". As you pointed out aiimpacts.org says "Median 10%", which seems like a mistaken notion.
Yes, am noobing and fumbling around a bit. I made the first edit hastily and immediately corrected before I had seen your reply. You are of course correct. I added a stike-through to show where my error lied.
Some aesthetic choices were made.
If you now got convinced that survival and future of everyone you know, including your own, depends on your immediate actions after reading this; you would not hesitate to put your daily tasks on halt to study and meditate on this text as if it was a live message being broadcast from an alien civilization. And afterwards you would not hesitate to drastically alter your life and begin doing what ever is in your power to steer away from the disaster. That is, IF you knew of such circumstance to any significant degree of certainty. Being simply told of such circumstance is not enough, because (simply put) world is full of uncertainties,... (read 1330 more words →)
Not sure if I understood correctly, but I think the first point just comes down to "we give AI a goal/goals" . If we develop some drive for instructing actions to an AI then we're still giving it a goal, even if it comes via some other program that tells it what those goals are at the moment in relation to whatever parameters. My original point was to contrast between AI having a... (read 499 more words →)