Summary : Whenever human beings seek to achieve goals far beyond their individual ability, they use leverage of some kind of another. Creating organizations to achieve goals is a very powerful source of leverage. However due to their nature, organizations are imperfect levers and the primary purpose is often lost. The inertia of present forms and processes dominates beyond its useful period. The present system of the world has many such imperfect organizations in power and any of them developing near-general intelligence without significant redesign of their utility function can be a source of existential risk/values risk.
When human beings seek to achieve large ambitious goals, it is natural to use some kind of leverage, some kind of ability to multiply one's power. Financial leverage, taking debt, is one of the most common means of leverage as it turns a small profit or spread into a large one. An even more powerful means of leverage is to bring people together and creating organizations to achieve the purpose that one had set out to achieve.
However, unlike the cleanliness of financial leverage(though not without problems of its own), using organizational leverage is messy, especially if the organization is created to achieve goals that are subtle and complex. There is an entire body of management literature that tries to align the interest of principals and agents. But most agents do not avoid Goodhart's law. As organizations grow, the mechanization of their incentive structures increases. Agents will do what they have been incentivized to do.
Recently, we saw an awesome demonstration of great futility, Quantitative Easing II. The purchase of a large number of US government bonds by the US Fed in the hope of creating more prosperity. People feeling poor due to the bursting of the housing bubble are definitely not the recipients of this money. And these are the people who eventually have to start producing and consuming again for this recession to end. Where would they attain the money from? The expected causal chain (econ experts can correct me if I'm wrong here) went like this
Buying US bonds - Creating the expectation of inflation in the market - Leading to banks wanting to lend out their reserves to people - leading to the people getting credit from banks - leading to them spending again - leading to improved profits in firms - leading to those firms hiring - leading to more jobs and so on.
The extremely long causal chain is not the main failure feature here. Nor is the fact that there are direct contradictory policies acting against step3 (paying interest on reserves maintained at the Fed) . My point is that even if this entire chain were to happen and the nominal end result, GDP growth, were to be achieved, the resulting prosperity would not be long lasting because it is not one that is based on a sustainable pattern of production and trade.
Maintaining equitable prosperity in a society that is facing competition from younger and poorer countries is a tough problem. Instead of tackling this problem head-on, the various governmental sources continued on their inertial paths and chose to adapt the patterns of home and asset ownership to create an illusion of prosperity. After all, the counter (GDP) was still running.
Many smart people have voiced opinions against such blind following of metrics in government, but almost all organizations, once beyond the grip of the founders fall into some such pattern. Compassionate mystic movements become rigid churches. Political parties for eg. pay more attention to lip service to issues, pomp, show and mind killing than to actual issues. Companies seek to make money at the expense of creating actual value, forgetting that money is only a symbol of value. A lot of people have bemoaned the brainpower that is moving into finance. And something even more repugnant, there is an entire economy thriving around the war on drugs, with everyone in on the cut.
In the short run, all these organizations/formations/coalitions are winning. So, voices against their behaviour that do not threaten them are being ignored.
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" - Upton Sinclair
There is a great deal of intelligence being applied today in these areas by people far more smarter than you or me. But the overall systems are still not as intelligent as a human yet. In the long run, they are probably undermining their own foundations.
The scary part really is these corporations and governments, which while being sub-humanly intelligent right now, are probably going to be at the forefront of creating GAI. I expect that near human intelligence will emerge in organizations and will most probably be a well knit human+computer team, probably with Brain Computer Interfaces. This team may or may not share all the values of the organization, but the incentives that this intelligence is being rewarded for, it will seek to achieve. And if these incentives are as narrowly phrased as
- maintain power over these certain set of people (As we might imagine the actual value system of government seems to be)
- make money without getting into legal trouble (As we might imagine the value system of a corporation would be)
then there will be a continuation of today's sheer insane optimization, but on a monstrous scale. The altruists amongst us have shown the inability to curb the sub-humanly intelligent larvae, what will we do the near human or super human butterfly? Competition between such entities would very quickly eliminate most compassionate values and a lot of what humanity holds dear. (My personal belief is that corporate AIs might be a little safer as they would probably crash the money system, but would not be lobbing nuclear weapons onto rivals).
In the end, I don't want to undermine the very idea of organizations because they have brought us unprecedented prosperity. I could not be transmitting my opinions to you without the support of many such organizations.
So, my take is that, I don't find the scary idea of SIAI as a completely alien idea. The present sources of optimization power, whether they be People in governments, LLCs in the present mixed economy system or political parties in the present plurality system, do not show any inclination towards understanding or moving towards true human morality. They do not "search their souls", they respond to incentives. They act like a system with a utility function, a function indifferent to morality. Their AIs will inherit these characteristics of these imperfect levers and there is no reason to expect, from increased intelligence alone, that the AI will move towards friendliness/morality.
EDIT : Edited to make clear the conclusion and set right the Goodhart's law link. Apologies to Vaniver, Nornagest, atucker, b1shop, Will_Sawin, AdShea, Jack, mwaser and magfrump who posted before the edit. Thanks to xamdam who pointed out the wrong link.