That is very strange. I associate the strategy of being upfront, honest, precise, and focused about AI extinction risk very strongly with ControlAI, second only to MIRI.
ControlAI's first campaign was about deepfakes. Maybe the only significantly indirect thing they've ever done, and it still wasn't what I would call deceptive. Ever since then, they have been focused on extinction risk, which can be clearly seen from their campaign statements signed by politicians. Their social media campaigns have all kinds of hooks about possible negative AI futures, but ...
We are much more in agreement than I expected! Thank you for clarifying. I agree with every point you made in this comment. (Excepting the comparison to continental philosophy, which is new to me and not something I think I can evaluate one way or the other.)
1The shape of superintelligence that we are building isn't necessarily the shape of superintelligence that will ultimately exist, though, right? That's one of the hard barriers to see beyond: the ability for an AI to invent new AI paradigms.
It's better than useless to study current systems if they give us insight into intelligence in general, and we might get useful information about the types of minds that are likely to exist. (I find work like the Natural Abstractions Hypothesis really interesting, and that benefits from empirical work). But that isn't quite the same as studying things that are shaped like the superintelligence we will get.
I think what you perceive as an aversion to empiricism and rigorous justification is just an acknowledgment of the areas where empiricism and rigorous justification are currently impossible.
It is very common for AI safety researchers to fall for the trap of doing mostly useless (or counterproductive) empirical work rather than trying to solve important problems. We know that something abstract like agent foundations will be necessary, because it is strictly impossible to do empirical work on superintelligence if superintelligence doesn't exist. (You probab...
1
1With a very short, messy pause, I expect humanity to be long gone before 2040. I am much more interested in the economic outlook of a decades-long or permanent moratorium, since that's the only version of the future we should expect to observe.
That the world is highly engineerable, which can lead to the relaxation or abolition of seemingly hard bottlenecks. Also that the world can respond extremely quickly to implement those changes when the incentives are right.
Overpopulation would have been a massive problem at different points in history if not for the invention of horseless transport and high-yield, resilient cereal crops. People living in New York City in the late 1800's or in developing nations in the 1960s and 70s were rescued from the worst hazards of overpopulation because of the motorc...
Here's the Metaculus data. If you click "Show Projection" on the graph, you'll see the trendline crossing "Metaculus Pro Forecasters" (which includes myself) in June 2027. That closely matches the current projection of ForecastBench, which puts the estimated LLM-superforecaster parity in May 2027. The closeness of the match may be coincidental to some degree, since the two running experiments use very different methods and targets, but I find the general agreement between these unrelated projects pretty compelling.
I asked the free versions of ChatGPT, Clau...
I don't understand the forecasting community's love of markets. Metaculus does just fine without a market system, and no one is destroying their lives using it.
Of course, now that the machine forecasting method has started, it will not take long to outstrip our feeble powers, so to speak. On the current trend, AI will be strictly superhuman at forecasting by mid next year, and the term "forecaster" may go the way of "computer," as a description of a machine rather than a human pursuit.
In that case, markets are again unnecessary overhead, compared to e.g. just asking 5 AI systems a couple times and taking the median prediction.
What kind of plays are there now, other than short-term?
I would really rather things come together long before the final hour, if we can at all help it. I would like to see that navigation take place over decades, if reality permits such sanity.
And while I do like imagining it, most people have failed to rise to the occasion so far, especially those in the AI companies. It's up to the general public now, to wake up the governments of the world and shut down the race.
My gut-check is to imagine a superintelligence that does the complicated acausal multiversal thing, and then separately imagine a superintelligence that just tries to take over the lightcone with no hesitation. Whichever ends up winning more stuff is the real one and has the right decision theory.
I just can't imagine a superintelligence being trolled by too-clever-by-half decision theory arguments, when it can just kill you and take your stuff (and the whole empty galaxy) instead. Maybe it has to modify the intergalactic probes so that it's provably the ki...
That sounds appealing if almost nothing ever changes, but we should expect discontinuities due to each or any of:
We just don't understand any of this stuff. Even a perfec...
The mechanism isn't [public disapproval] -> [AI moratorium]. That sounds hopeless to me, too. What I'm actually relatively optimistic about is [public awareness] -> [educating policymakers] + [signal to policymakers that they are allowed to do something about it] -> [common knowledge of the desire to act] -> [AI moratorium]. This feels very doable to me after getting back from speaking with my congressional offices in DC.
The key here is that most politicians don't want to take these risks either. Really very few people are on board with the who...
You can pursue both goals, but you have chosen not to, and I find that alarming.
There are many, many concrete accusations in this post. It doesn't particularly matter whether there is a "main" one. Vassar is a serial psychological and sexual abuser, and has caused multiple deaths. That is bad.
Plex opening up about his experiences with this abuser was a terrifying thing to do. The appropriate response in this circumstance is to respond with empathy. If you aren't going to do that, it is better not to say anything at all. I understand that we are on LessWron...
The rationalist community makes itself vulnerable to bad actors when it lacks the ability to just "be normal about things" and strongly reject bad behavior.
It's great to ask clarifying questions and weigh evidence appropriately. But the central premise here isn't in doubt. If survivors speak out and the go-to response is to nitpick the way in which they communicated about the harm, that just tells me you aren't safe people.
It sounds like he doesn't have any interest in saving the world at all. He is playing a completely different game from the people who actually value humans.
Do you think it is more important to dissect the way that people communicate, or to protect vulnerable people from predators?
My takeaway from this post is that an exceptionally evil man has caused significant harm to the rationalist community, leading directly to deaths and damaging the community's ability to do extremely important work. Is that not your primary takeaway?
On tractability:
I am in Washington DC today and will speak with the offices of both of my senators tomorrow, with 3 others also from Arizona, to educate them on the issue and demand that they call for a global agreement to ban the creation of superintelligent AI. 50+ others are doing the same thing for their state on the same day.
My representative Greg Stanton already (quietly for now) supports an ASI ban, primarily due to my personal efforts to educate him on the topic. My state-level representative Stacey Travers introduced an AI safety transparency bill...
If you model fighting with Anthropic as "infighting", we have very different ideas of what people and organizations are acceptable to associate with. Anthropic is doing an extraordinarily evil thing by trying to create a superintelligence. To the degree that there are "sides" anywhere, they are approximately maximally not on my side.
I strongly agree with your perspective on what high integrity and honesty entail!
I have been in the PauseAI movement since August 2023, and I cannot recall ever observing PauseAI Global or PauseAI US as organizations behaving recklessly in regard to the truth. We inherited a lot of rationalist obsession with honesty, which has predominantly been a very good thing. Holly's alleged recklessness in point of fact does not bleed over to PauseAI US's communications with the public or policymakers.
Holly as an individual has personal beef with many people in the r... (read more)