All of Roko's Comments + Replies
It certainly seems that a mastery of tank warfare would have helped a lot. But the British experience with tanks shows that there was a huge amount of resistance within the military to new forms of warfare. Britain only had tanks because Winston Churchill made it his priority to support them.
New weapon systems are not impressive at first. The old ways are typically a local optimum. So the real question here is how to leave that local optimum!
I'm struggling to see why fun books would make any difference. Germany didn't lose because it ran out of light reading material.
As for troop morale and so on, I don't think that was a decisive element as by the time it started to matter, defeat was already overdetermined.
In other words, I think Germany would have lost WWI even with infinite morale.
If it pays out in advance it isn't insurance.
A contract that relies on a probability to calculate payments is also a serious theoretical headache. If you are a Bayesian, there's no objective probability to use since probabilities are subjective things that only exist relative to a state of partial ignorance about the world. If you are a frequentist there's no dataset to use.
There's another issue.
As the threat of extinction gets higher and also closer in time, it can easily be the case that there's no possible payment that people ought to rationally accept....
Even in a traditional accounting sense, I'm not aware that there is any term that could capture the probable existential effects of a research, but I understand what @So8res is trying to pursue in this post which I agree with. But, I think apocalypse insurance is not the proper term here.
I think IAS/IFRS 19, actuarial gains or losses / IFRS 26 Retirement benefits are more closer to the idea - though these theortical accounting approaches applies to employees of a company. But these can be tweaked to another form of accounting theory (on another form ...
I think you have a nerdy novel society and a loss of WWI for the same reasons it was lost in our timeline
Sure you can bring decision theory knowledge. All I'm disallowing is something like bringing back exact plans for a nuke.
Well, it turned out that attacking on The Western Front in WWI was basically impossible. The front barely moved over 4 years, and that was with far more opposing soldiers over a much wider front.
So the best strategy for Germany would have been to dig in really deep and just wait for France to exhaust itself.
At least that's my take as something of an amateur.
But the British could have entered the war anyway. After all, British war goals were to maintain a balance of power in Europe and they don't want France and Russia to fall and Germany to be too strong.
OK, but if I am roleplaying the German side, I might choose to still start WWI but just not attack through Belgium. I will hold the Western Front with France and attack Russia.
I think violence helps unaligned AI more than it helps aligned AI.
If the research all goes underground it will slow it down but it will also make it basically guaranteed that there's a competitive, uncoordinated transition to superintelligence.
It seems that I was mostly right in the specifics, there was a lot of resistance to getting rid of Altman and he is back (for now)
Well the new CEO is blowing kisses to him on Twitter
Well the board are in negotiations to have him back
"A source close to Altman says the board had agreed in principle to resign and to allow Altman and Brockman to return, but has since waffled — missing a key 5PM PT deadline by which many OpenAI staffers were set to resign. If Altman decides to leave and start a new company, those staffers would assuredly go with him."
"A source close to Altman" means "Altman" and I'm pretty sure that he is not very trustworthy party at the moment.
I think there's a pretty big mistake here - the value of not getting flu is a lot more than $200.
At a $5M value of life, each day is worth about $200, so 7 days of almost complete incapacitation is -$1400.
I would certainly pay $1400 upfront to make a bad flu just instantly stop.
dath ilan is currently getting along pretty well without AGI
I hate to have to say it, but you are generalizing from fictional evidence
Dath ilan doesn't actually exist. It's a fantasy journey in Eliezer's head. Nobody has ever subjected it to the rigors of experimentation and attempts at falsification.
The world around us does exist. And things are not going well! We had a global pandemic that was probably caused by government labs that do research into pandemics, and then covered up by scientists who are supposed to tell us the truth about pandemics. THA...
1
1Yes, and I believe that the invention and spread of firearms was key to this as they reduce the skill dependence of warfare, reducing the advantage that a dedicated warband has over a sedentary population.
What happened in the 1200s is that Mongols had a few exceptionally good leaders
It's consistent with the overhang model that a new phase needs ingredients A, B, C, ... X, Y, Z. When you only have A, ... X it doesn't work. Then Y and Z come, it all falls into place and there's a rapid and disruptive change. In this case maybe Y and Z were good leaders or something. I don't want to take too strong a position on this, as given my research it seems there is still debate among specialists about what exactly the key ingredients were.
Most people, ultimately, do not care about something that abstract and will be happy living in their own little Truman Show realities that are customized to their preferences.
Personally I find The World to be dull and constraining, full of things you can't do because someone might get offended or some lost-purposes system might zap you. Did you fill in your taxes yet!? Did you offend someone with that thoughtcrime?! Plus, there are the practical downsides like ill health and so on.
I'd be quite happy to never see 99.9999999% of humanity ever again, to simpl...
Could an alien observer have identified Genghis Khan's and the Mongol's future prospects
Well, probably not to that level of specificity, but I think the general idea of empires consuming vulnerable lands and smaller groups would have been obvious
Well, sometimes they can, because sometimes the impending consumption of the resource is sort of obvious. Imagine a room that's gradually filling with a thin layer of petrol on the floor, with a bunch of kids playing with matches in it.
One possible way to kill humans
I suspect that drones + poison may be surprisingly effective. You only need one small-ish facility to make a powerful poison or bioweapon that drones can spread everywhere or just sneak into the water supply. Once 90% of humans are dead, the remainder can be mopped up.
Way harder to be able to keep things running once we're gone.
This post is way too long. Forget clown attacks, we desperately need LLMs that can protect us from verbosity attacks.
You can just create personalized environments to your preferences. Assuming that you have power/money in the post-singularity world.
a technical problem, around figuring out how to build an AGI that does what the builder wants
How does a solution to the above solve the coordination/governance problem?
Ah, I see. Yeah, that's a reasonable worry. Any ideas on how someone in those orgs could incentivize such behavior whilst discouraging poorly thought out pivotal acts?
the fact that we are having this conversation simply underscores how dangerous this is and how unprepared we are.
This is the future of the universe we're talking about. It shouldn't be a footnote!
researchers at big labs will not be forced to program an ASI to do bad things against the researchers' own will
Well these systems aren't programmed. Researchers work on architecture and engineering, goal content is down to the RLHF that is applied and the wishes of the user(s), and the wishes of the user(s) are determined by market forces, user preferences, etc. And user preferences may themselves be influenced by other AI systems.
Closed source models can have RLHF and be delivered via an API, but open source models will not be far behind at any given p...
AI researchers would be the ones in control
No. You have simplistic and incorrect beliefs about control.
If there are a bunch of companies (Deepmind, Anthropic, Meta, OpenAI, ...) and a bunch of regulation efforts and politicians who all get inputs, then the AI researchers will have very little control authority, as little perhaps as the physicists had over the use of the H-bomb.
Where does the control really reside in this system?
Who made the decision to almost launch a nuclear torpedo in the Cuban Missile Crisis?
I would question the idea of "control" being pivotal.
Even if every AI is controllable, there's still the possibility of humans telling those AIs to bad things and thereby destabilizing the world and throwing it into an equilibrium where there are no more humans.
Global compliance is the sine qua non of regulatory approaches, and there is no evidence of the political will to make that happen being within our possible futures unless some catastrophic but survivable casus belli happens to wake the population up
Part of why I am posting this is in case that happens, so people are clear what side I am on.
Well, the AI technical safety work that's appropriate for neural networks is about 5-6 years old, if we go back before 2017 I don't think any relevant work was done
Conversely, if we had a complete technical solution, I don't see why we necessarily need that much governance competence.
As I said in the article, technically controllable ASIs are the equivalent of an invasive species which will displace humans from Earth politically, economically and militarily.
"If the world were unified around the priority of minimizing global catastrophic risk, I think that we could reduce risk significantly further by implementing a global, long-lasting, and effectively enforced pause on frontier AI development—including a moratorium on the development and production of some types of computing hardware"
This really needs to be shouted from the rooftops. In the public sphere, people will hear "responsible scaling policy" as "It's maximally safe to keep pushing ahead with AI" rather than "We are taking on huge risks because politicians can't be bothered to coordinate".
This really needs to be shouted from the rooftops.
I disagree. I think it's important that we shout from the rooftops that the existential risk from AI is real, but I disagree that we should shout from the rooftops that a sufficiently good pause would solve it (even though I agree with Paul that it is true). I talk about this in this comment.
Historically, I think that a lot of causes have been hurt by a sort of purity-testing where scientists are forced to endorse the most extreme policy, even if it's not the best policy, on the idea that it would solve ...
6
2The problem with a naive implementation of RSPs is that we're trying to build a safety case for a disaster that we fundamentally don't understand and where we haven't even produced a single disaster example or simulation.
To be more specific, we don't know exactly which bundles of AI capabilities and deployments will eventually result in a negative outcome for humans. Worse, we're not even trying to answer that question - nobody has run an "end of the world simulator" and as far as I am aware there are no plans to do that.
Without such a model it's very diff...
You need a method of touching grass so that researchers have some idea of whether or not they're making progress on the real issues.
We already can't make MNIST digit recognizers secure against adversarial attacks. We don't know how to prevent prompt injection. Convnets are vulnerable to adversarial attacks. RL agents that play Go at superhuman levels are vulnerable to simple strategies that exploit gaps in their cognition.
No, there's plenty of evidence that we can't make ML systems robust.
What is lacking is "concrete" evidence that that will result in blood and dead bodies.
This is where I'd like to insert a meme with some text like "did you even read the post?" You:
- Make a bunch of claims that you fail to support, like at all
- Generally go in for being inflammatory by saying "it's not a priority in any meaningful value system" i.e. "if you value this then your system of meaning in the world is in fact shit and not meaningful"
- Pull the classic "what I'm saying is THE truth and whatever comes (the downvotes) will be a product of peoples denial of THE truth" which means anyone who responds you'll likely just point to and say someth
you only start handing out status points after someone has successfully demonstrated the security failure
Maybe you're right, we may need to deploy an AI system that demonstrates the potential to kill tens of millions of people before anyone really takes AI risk seriously. The AI equivalent of Trinity.
It's not just about "being taken seriously", although that's a nice bonus - it's also about getting shared understanding about what makes programs secure vs. insecure. You need a method of touching grass so that researchers have some idea of whether or not they're making progress on the real issues.
Do you just like not believe that AI systems will ever become superhumanly strong? That once you really crank up the power (via hardware and/or software progress), you'll end up with something that could kill you?
Read what I wrote above: current systems are safe because they're weak, not safe because they're inherently safe.
Security mindset isn't necessary for weak systems because weak systems are not dangerous.
I believe the security mindset is inappropriate for AI
I think that's because AI today feels like a software project akin to building a website. If it works, that's nice, but if it doesn't work it's no big deal.
Weak systems have safe failures because they are weak, not because they are safe. If you piss off a kitten, it will not kill you. If you piss off an adult tiger...
The optimistic assumptions laid out in this post don't have to fail in every possible case for us to be in mortal danger. They only have to fail in one set of circumstances that someone ...
Yes, and also using radio with good encryption to communicate quickly.