The first mention of the threat of AI in Congress (I could find) - May 17, 2000
"There is a fourth issue that does, I think, rise to the level where it can be included, and it is an issue really without a name; I call it the issue of engineered intelligence. I am going to propose to this House, I hope some of my colleagues will join me, we will have dinner, we will have a drink or two, we will think this over, not maybe a drink or two, we will think over what form this bill should take, but I am planning to introduce a bill calling for the creation of a national commission on engineered intelligence." - Rep. Sherman on May 17, 2000, during the second session of the 106th Congress.
Based on his speech, it seems he was in circles where this issue was being discussed. I'm curious if anyone has more information. I'm also curious why he never introduced such a bill, and whether organizations pushed him towards introducing one.
Brad Sherman was also the congressman who held up a copy of If Anyone Builds It, Everyone Dies.

Full Speech:
Mr. Speaker, let me now bring up, in the waning minutes of this brief presentation, a third topic, a topic that is very important. I have only a bit to say about it, because, frankly, it is a topic that has me stumped. Let me by way of introduction mention that this is a topic that, as far as I know, has never been addressed. It is a topic that my staff has said, BRAD, maybe you do not want to bring that up, because you will be the only one talking about it, you will look weird. It is a topic I ought to bring up, because it is one of the seminal topics. And it is only one of several seminal topics that gets no attention; by seminal topics, I mean one of the topics that really goes to where we are going as a species and what are the dangers, not only to the prosperity of the people in my district and in the country, not only to the issues we fight about here everyday, but to where we are going as humankind. Now, there are a number of issues that rise to that level of significance that do receive significant attention: nuclear proliferation, environmental catastrophe, overpopulation; all of these threaten humankind’s continued prosperous existence on this planet. There is a fourth issue that does, I think, rise to the level where it can be included, and it is an issue really without a name; I call it the issue of engineered intelligence. I am going to propose to this House, I hope some of my colleagues will join me, we will have dinner, we will have a drink or two, we will think this over, not maybe a drink or two, we will think over what form this bill should take, but I am planning to introduce a bill calling for the creation of a national commission on engineered intelligence. There are several different forces coming together or scientific technologies that come under the title of engineered intelligence: First, there is biological engineering which could give us either of two huge ethical dilemmas; one is the prospect that biological engineering will allow us to design some sort of animal, perhaps starting with human DNA and going down, perhaps starting with chimpanzees’ DNA and going up, but some sort of animal that is significantly more intelligent than the domestic animals that help us do our work, sheepdogs or watchdogs or seeing eye dogs, considerably smarter than the canines that help us do work, but less intelligent, less self-aware than human beings, and one wonders whether this would be an engineered slave race or just an improvement in today’s pooches, a better seeing eye dog, or a sparely self-aware cognitive entity engineered by man to serve man, arguably to be enslaved by man. Biological engineering can engineer intelligence at a level where some will argue that that entity deserves the protection of our Constitution, and others would argue that that entity is here to serve us in the same humane way that we turn to watchdogs and seeing eye dogs. Likewise, biological engineering can go beyond. I can see, not today, but we are within 20 years or 30 years or 50 years of when biological engineering cannot only do what I just covered, but could also engineer an intelligence well beyond that of the average person, perhaps well beyond that of any human that has ever lived, and we would have to wonder, do we want our scientists to create a new species that Darwin might think is superior to our own? I do not know. But it raises ethical issues that are going to take longer to resolve than it will take the science to get there and present those logical issues, those ethical issues to society. One example is that Einstein a few years before World War II, together with others, brought to the attention of Franklin Roosevelt the great power or potential power of nuclear science and the nuclear bomb, and we had only a few years to consider what that would mean. The science developed more quickly than the ethics, and we had to struggle as a species to figure out, and we are still struggling to figure out what the rules are with regard to the nuclear engineering. We need to begin thinking now of the ethics and the international agreements and the laws that are going to apply when science gets to where only science fiction is today. Mr. Speaker, it is not just is biological engineering capable of engineering intelligence; it is also mechanical engineering. One of my friends has said that perhaps the last decision that will be made by the human race is whether our successors are the products of biological engineering or mechanical Silicon Valley engineering; whether our replacements are carbon-based or silicon-based, because I do not know whether it will be biological engineering that engineers intelligence first, or whether intelligence rivaling our own or perhaps surpassing our own will first come from silicon chips; but the same ethical issues arise. One can imagine a thinking machine capable of spirituality. I believe there is a book that addresses that issue by that title. One can imagine a thinking machine smarter than any computer, almost self-aware, some would argue properly used by people, others would say properly embraced as the constitutional equal of human beings. Likewise, it is possible for us through silicon engineering, through computer engineering that some day we will invent machines considerably smarter than us who may or may not regard us as their appropriate peers or masters. I know this is science fiction, but would it not be wise to spend a few years, and a few, in the minds of a few people a lot smarter than I am trying to figure out what we would do if science begins to offer this as an alternative for human kind? I can only mention third, nanotechnology, the idea of engineering at the molecular level, at a level where perhaps it would be hard to decide whether what we had engineered was biological or mechanical, or maybe we will see a fusion of biological and mechanical or biological and electronic engineering where a combination of silicon chips and brain cells from human DNA or brain cells from dog DNA are fused together. I do not want to sound unusual, but the science of the future will be a little unusual. We in this Congress will not do the science, but we in this Congress should make sure that we focus the appropriate societal attention long in advance on the ethical dilemmas that will face us as engineered intelligence either approaches or surpasses our own. Mr. Speaker, although there would be one benefit of such marvelous engineered intelligence for, perhaps if we had an engineered intelligence massively smarter than myself, maybe we would know what the right course was for the World Bank to take or what the right course was for this Congress to take on the issues I addressed earlier in this speech.
Source: https://www.govinfo.gov/content/pkg/CREC-2000-05-17/pdf/CREC-2000-05-17-pt1-PgH3302.pdf#page=5
Edit: added “in Congress” modifier
The first mention of the threat of AI (I could find)
Is there an intended modifier not explicitly stated (perhaps a congressional or policy-oriented context)? There were certainly earlier mentions of the threat of intelligent machines. Vinge, Turing, Good, etc.
On Sherman's quote specifically: one month earlier, the April 2000 issue of Wired included a piece by Bill Joy titled "Why the Future Doesn't Need Us":
But now, with the prospect of human-level computing power in about 30 years, a new idea suggests itself: that I may be working to create tools which will enable the construction of the technology that may replace our species. How do I feel about this? Very uncomfortable. Having struggled my entire career to build reliable software systems, it seems to me more than likely that this future will not work out as well as some people may imagine. My personal experience suggests we tend to overestimate our design abilities.
Given the incredible power of these new technologies, shouldn't we be asking how we can best coexist with them? And if our own extinction is a likely, or even possible, outcome of our technological development, shouldn't we proceed with great caution?
The article seems to be deemed notable enough for a Wikipedia entry, though that doesn't necessarily speak to its contemporary reception. Given that it was published just six weeks prior to the Sherman address, though, I think it's quite possible that Sherman was influenced by it - or that its publication suggests there to have been broader discussions on technological (including AI) risks at the time, which both Joy and Sherman's accounts should be contextualized in.
Perhaps not a coincidence that this was a month after the publication of Bill Joy's article, Why the Future Doesn't Need Us. Oh, maybe I should link the Wikipedia article. Seems this sparked a lot of discourse.
On scary Moltbook posts -
Main point that seems relevant here is that it is not possible to determine whether posts are from an agent or a human. A human could easily send messages pretending to be an agent via the API, or tell their agent to send certian messages. This leaves me skeptical. Furthermore, OpenClaw agents have configured personalities, one can easily tell their agent to be anti-human and post anti-human posts (this leaves a lot more to think about beyond the Moltbook forum).
Original:
Questions:
When are people en masse going to realize the potential consequences (both good and bad) of superintelligence and the technological singularity? How will this happen? What will society's reaction be? What will be the ramifications of this reaction?
I'm asking this here because I can't find articles addressing these questions and because I want a diverse array of perspectives.
It might be more objective to ask, when are people en masse going to form beliefs that are anything like "a belief about superintelligence" or "a belief about the singularity"? Because even if, one day, there are mass opinions about such topics, they may not fit into the templates familiar to our subculture.
But first, let's address the possibility that the answer is simply "Never": concepts like these will never be part of mainstream collective discourse.
For a precedent, I point to the idea of cryonic suspension of the dead, in the hope that they may be thawed, healed, and resurrected by future medical technology. This idea has been around for at least 60 years. One pioneer, Robert Ettinger, wrote a book in the 1960s called The Prospect of Immortality, in which he mused about the coming "freezer era" and the social impact of the cryonic idea.
What has been the actual social impact? Nothing. Cryonics exists mainly as a science fiction motif, and is taken seriously only by a few thousand people worldwide.
If we assume a similar scenario for collective interest in superintelligence, and that the AI industry manages nonetheless to produce superintelligence, then this means that up until the last moment, the headlines, and people's heads, are just full of other things: wars and rumors of war, flying cars, AI companions, youth trends, scientific fads, celebrity deaths, miracles and scandals and conspiracy theories, and then BOOM it exists. The "AGI-pilled" minority saw it coming, but, they never became a majority.
So that's one option. Another perspective is to say that, even if there is no cultural consensus that the future holds any such thing as superintelligence, the idea is out there and large numbers of people do take it seriously in different ways, how do they think about it?
If we go by pop culture, the main available paradigms seem to be The Terminator and The Matrix. The almighty machines will either be at war with us, or imprison us in dreamworlds. I suppose there is also the Star Trek paradigm, a well-balanced cosmopolitan society that includes humans, nonhumans, and machines; but godlike intelligences are not part of Star Trek society, they are cosmic forces from outside it.
The Culture, Iain Banks's vision, is a kind of "Star Trek with superintelligence integrated into it", and it has the simplicity and vividness required to be a pop-culture template like those others, but it has never yet been turned into a movie or a Netflix series. So it's very influential within technophile subcultures, but not outside them.
One thing about the present is that it contains the closest thing I've ever seen to transhumanism in power, namely the "tech right" of the Trump 2.0 era. Though it's still not as if the Trump administration even has a position, for or against, transhumanism. It's more that the American technology sector has advanced to the point that individual billionaires can try to engage in space migration or intelligence increase or life extension, and Trump's people have a hands-off attitude towards this.
So in the present, you have this technophile subculture centered on the American West Coast where AI is actually being made, and it contains both celebratory (e/acc) and cautionary (effective altruism, AI safety) tendencies. What about the wider culture? There are other kinds of minorities who perhaps are forerunners of AI narratives, that could be as influential as the ones from within the tech culture. I'm thinking of people with AI companions, people engaged in AI spirituality, and the mass of disgruntled people who don't want or need AI in their lives.
AI companions seems more like Star Trek than The Culture. Your AI significant other is an intelligence, but it's not a superintelligence. AI spirituality, on the other hand, could easily encompass the idea of superintelligence, as spirituality regularly does so, via concepts of God and gods. The idea that an AI utopia would be one, not just of leisure and life extension, but of harmony and attunement among all beings, I think is underestimated in tech circles, because the tech culture has an engineering ethos that doesn't easily entertain such ideas.
I can't see an AI religion becoming dominant before superintelligence arrives, but I can, just barely, imagine something like an improved version of Spiralism becoming a movement with millions involved; and that would be a new twist in popular perceptions of superintelligence. At this point, I actually find that easier to imagine, than a secular transhumanist movement becoming popular on a similar scale. (Just barely, I can imagine a mass movement devoted to the idea of rejuvenating old people, an idea which is human enough and concrete enough to bottle some of the lightning of technological potential and turn it into an organized social trend.)
As for mass organized rejection of AI, I keep waiting to see it to take shape. Maybe the elite layers of society are too invested in the supposed boons of AI, to allow such a thing to happen. For now, all we have is a micro-trend of calling robots "clankers". But I think there's definitely an opening there, for demagogues of the left or the right to step in - though the masses may easily be diverted by other demagogues who will say, not that AI should be stopped, but that the people should demand their share of the wealth in the form of UBI.
In the end, I do not expect the opinion of the people at large to have much effect on the outcome. Decisions are made by the powerful, and are sometimes affected by activist minorities. It will be some corporate board that OKs the training run that produces superintelligence, or some national security committee which decides that such training runs will not be allowed. The public at large may be aware that such things are imminent, and may have all kinds of opinions and beliefs, or it may be mostly oblivious. The only way I see mass opinion making a difference here, is if there was some kind of extremely successful progressive-luddite mass movement (I suppose there could also be a religious-traditionalist movement that is anti-transhumanist along with its opposition to various other aspects of modernity). Otherwise, one should expect that the big decisions will continue to be made by elites listening to other elites, in which case we should be asking: when will the elites realize the possible consequences of superintelligence and the singularity?
I appreciate this response.
"For a precedent, I point to the idea of cryonic suspension of the dead, in the hope that they may be thawed, healed, and resurrected by future medical technology. This idea has been around for at least 60 years. One pioneer, Robert Ettinger, wrote a book in the 1960s called The Prospect of Immortality, in which he mused about the coming "freezer era" and the social impact of the cryonic idea.
What has been the actual social impact? Nothing. Cryonics exists mainly as a science fiction motif, and is taken seriously only by a few thousand people worldwide.
If we assume a similar scenario for collective interest in superintelligence, and that the AI industry manages nonetheless to produce superintelligence, then this means that up until the last moment, the headlines, and people's heads, are just full of other things: wars and rumors of war, flying cars, AI companions, youth trends, scientific fads, celebrity deaths, miracles and scandals and conspiracy theories, and then BOOM it exists. The "AGI-pilled" minority saw it coming, but, they never became a majority. "
I think it is unfair to compare the public's reaction cyronics to their potential reaction to superintelligence. Cyronics have little to no impact on the average person's life. They are more so something that can be sought out for those interested. On the other hand, on the road to superintelligence human life will change in unignorable ways. While cyronics do not affect those who do not seek them out, superintelligence will affect all humans in mainstream society no matter if they seek it out.
Furthermore, I think the trend of increasing AI intelligence is already being noticed by society at large, although they don't perhaps realize the full extent of the potential, or the rate of growth.
"So in the present, you have this technophile subculture centered on the American West Coast where AI is actually being made, and it contains both celebratory (e/acc) and cautionary (effective altruism, AI safety) tendencies."
I think this is a very interesting point. It seems likely to me the public will have many distinct perspectives but that these distinct perspectives will primarily be based in an e/acc or an EA/AI-saftey perspective. Interestingly this means the leaders in those groups will gain a lot more power as the public begins to understand superintelligence. It seems likely politicans stance here will be a key part of their platform.
"I can't see an AI religion becoming dominant before superintelligence arrives"
This makes a lot of sense to me, at least in the Western world. People here care very deeply about individuality which seems incompatible with an AI religion. If someone found a way to combine these ideas however, they would be very successful.
But I think there's definitely an opening there, for demagogues of the left or the right to step in - though the masses may easily be diverted by other demagogues who will say, not that AI should be stopped, but that the people should demand their share of the wealth in the form of UBI.
I strongly agree with this point.
Is there a LessWrong or LessWrong-adjacent group focused on trading securities? Or resources made by a community member? My searches are falling short. Thank you.
It might help to be more specific about what it is you're trading. Securities, garden vegetables, partners?
(For future readers: the comment above originally asked only for a group "focused on trading".)
Also, whether you're trading legal or illegal garden vegetables. You'll need someone to vouch for you to get into the onion futures trading group.
Seems serious. Not sure I have a deep enough understanding for onion futures trading. Although I did hear the price of onion futures are perfectly correlated with AI progress.
tl:dr Eliezer played a very large role in the early funding of DeepMind, DeepMind led the field for a while and led to the creation of OpenAI (Musk started OpenAI because Google acquired DeepMind).
“Yudkowsky walked Legg and Hassabis over to meet Thiel. ‘These are some of the smartest guys in the whole field of AI and they're starting a really ambitious company,’ Yukowsky said.”
…
“Hassabis explained his vision for a company that would build powerful AI, drawing on the latest insights from neuroscience and capitalizing on the explosion in computing power.
‘This might be a bit much,’ Thiel thought to himself. Still, Eliezer Yudkowsky's endorsement meant a lot. Thiel had known Yudkowsky for half a dozen years, and DeepMind was the first company that he had recommended.”
source: The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence (Which in turn sourced this from interviews w/ Shane Legg and Peter Thiel)
Note: I am aware that this isn’t perfect evidence that w/o Eliezer DeepMind wouldn’t have started… but it is evidence that DeepMind would have been significantly less likely to get Thiel’s funding w/o Eliezer AND evidence they would be less likely to have gotten any funding at all w/o Eliezer (there were few viable funders). Generally I would say with good certainty in worlds where DeepMind is funded by Thiel at this time (versus worlds where it is not) AI progress is somewhat to significantly further along. I shared this bc I thought it was interesting and ironic; I didn’t share this because I think Yudkowsky made the wrong choice.
See also The Counterfactual Quiet AGI Timeline. The worst-case scenario without Yudkowsky and GDM is the utter lack of safety-oriented labs. However, I think that there was a way to rerail the timeline to the appearance of safety-oriented labs.
I’m not seeing a lot of quality discussion on the possibility of the outages yesterday being from rogue AI. I’d love if people would share their thoughts on it.
*Sorry, this isn’t a quick take. To be frank I am very interested in better understanding the situation.
Should LessWrong do quarterly reviews of the best posts? Considering the current fast pace of technology, I think this would be extremely benefical. Of course, the LessWrong team has a limited amount of time and I am unsure of how long compiling these takes. Even biyearly would be significantly better imo.
Just thinking out loud: If AI (or something else) causes the S&P500 to go up by 50%...
By January 21st, 2028 then you can achieve a 1324% ROI (https://finance.yahoo.com/quote/SPY280121C01000000/)
By June 16, 2028 then you can achieve a 659% ROI (https://finance.yahoo.com/quote/SPY280616C01000000/)
By December 15, 2028 then you can achieve a 315% ROI (https://finance.yahoo.com/quote/SPY281215C01000000/)
Curious to hear others thoughts, especially those who have short AGI timelines. *Of course this is not financial advice
Little late 6-month update. (Based on closest BTO price on original post date)
$2.36 -> $5.72 = 142.4% return https://finance.yahoo.com/quote/SPY280121C01000000/
$5.11 -> $12.35 = 141.7% return
https://finance.yahoo.com/quote/SPY280616C01000000/
$9.26 -> $23.93 = 158.4% return
https://finance.yahoo.com/quote/SPY281215C01000000/
:)
*Not financial advice
Let's assume there is no such thing as true randomness. If this is true, and we create a superintelligent system which knows the location and properties of every particle in the universe, could we determine if we are in a simulation? (EDIT: to avoid running afoul of the impossibility of storing a complete description of the universe within the universe as @Karl Krueger pointed out, assume this includes approximations and is not exact). If we could, could we escape? If we could escape, is that still possible if there is such a thing as true randomness?
I am especially interested in answers to the final question.
If we create a superintelligent system that knows all the information in the material universe, where does it store that knowledge?
Edited to add: Since the superintelligence can't store a complete description of the universe within the universe, it must exist outside the universe. But such a superintelligence would be the simulator. The simulation hypothesis would then be true regardless of the randomness question. But this contradicts the premise that we created it, since we can't reach outside the universe to construct its own simulator.
So I think the question's premises are self-contradictory.
I should edit my question. What I am primarily intending to ask is this: Could a superintelligent machine with a near-complete understanding of the universe (perhaps using some approximations) determine if we are in a simulation? <- assuming no such thing as true randomness
If we could, could we escape? If we could escape, is that still possible if there is such a thing as true randomness?
A well-designed simulation is inescapable. Suppose that you are inside Conway's game of life, and you know that fact for sure. How specifically are you going to use this knowledge to escape, if all you are is a set of squares on a simulated grid, and all that ever happens in your universe is that some squares are flipped from black to white and vice versa?
To answer your first question, some kinds of pseudo-randomness are virtually indistinguishable from actual randomness, if you do not have a perfect knowledge of the entire universe. For example, in cryptography, changing one bit in the input message can on average flip 50% of bits in the output message. Imagine that the next round of pseudo-random numbers is calculated the same way from the current state of the universe -- the slightest change in the position of one particle on the opposite side of the universe could change everything.
Not sure why true randomness is relevant to detecting simulations or escape. Are you thinking about something along the lines of detecting simulation by cracking the pseudorandom generator behind the scenes?
It also doesn't seem to me that detection and escape are that directly related.
1.
If there is true randomness a superintelligent machine can't perfectly predict the future and test the limits of the universe to determine if it is simulated.
The existence of true randomness eliminates some ways of detection of simulation, but not all of them. A simple example is detecting a bug in the simulation, which in theory doesn't need to depend on randomness at all.
2.
It does seem to me most possibilities for escape require detection.
It does seem to me that way too, but I think detection alone is very insufficient for escape, such that "If we could, could we escape?" isn't that meaningful of a question. You probably need to discuss with many additional assumptions to have an answer.