Thank god for slow takeoff.
So far!
1I think they'll release an underwhelming product that is neither superintelligent nor safe, mostly as a bid for relevance.
I don't think Ilya is any better than Sam, Dario, Demis, or Elon. They're all AI company CEOs, which is a morally compromising thing to be. They're all trying to build superintelligence, which is a very evil thing to try to do.
6
3I've been saying for some time (out loud to my friends, anyway) that deduction is just a thousand inductions in a trenchcoat. I take this to be true in an abstract epistemological sense, but perhaps more importantly in the sense of the physical operation of our brains. It's obvious to me that the human brain doesn't contain a formal logic module. I don't see why we should expect AI to be any different (aside from "it's in a computer, and I think I know what computers are").
But if you imitate how a human with certain emotions would act you are not having those emotions, you are just acting.
Certainly, but it turns out that LLMs do appear to functionally have emotions. Whether they feel emotions is an open question, but they do seem to have them.
Do I get Bayers Points?
Relatedly, if someone you know is accused of acting poorly toward another person, the observation that they never act that way toward you may provide nearly zero Bayesian evidence in their favor.
A very common example:
Outside of the specific example, and I'm not sure if you meant it this way, but I would caution against using "the world as we know it might end soon" as a reason to make decisions that would be bad if the world somehow stayed mostly normal.
To the specific example, I would search (or ask an LLM) for expert sources on raccoon behavior and welfare, and run with their conclusions.
Not that I know of, and I would also like to see such a report. It matches the angle I took in my contributions to the Metaculus Labor Hub.
but like many movements, the passionate, smart people are attracted into unproductive directions because they stick to what's memetic, avoid conflicting with allies...
I was genuinely confused for a moment while reading this, because this is a perfect description of the pro-industry, anti-pause arm of rationalism and EA. Everyone working for an AI company (or their proxies like Coefficient Giving), or trying to "align" LLMs, or eschewing protests, or refusing to speak to politicians about shutting down the unconscionable suicide race... they are getting ...
1"ASI Platform Provider" is a deeply cursed phrase.
1Focusing on choices that essentially affect only the chooser:
"If you've never been mugged, you're taking too few alleyway shortcuts" would be terrible advice, whereas "if you've never been turned down for a date, you're not putting yourself out there enough / aiming high enough" is generally good advice. So now we are talking price.
Setting aside that doing things that are bad for ourselves is often bad for others, and looking at the specific example offered: To me, missing a flight would be like getting mugged. Or at least, it would be much more like getting mugged than getting romantically rejected by a stranger. If it's a thing that happens once in my entire life, then I haven't spent enough time at airports.
In some cases, risks you impose upon yourself become risks imposed upon others. Generally advocating for people to impose more risks upon themselves poses additional risks on everyone. I don't want to be in an airport filled with people panicking about being late for their flight. That sounds terrible. And I definitely don't want to be driving on a road where everyone is significantly more likely to be late. That's more dangerous.
I'm wrestling through the idea now, though, that it may not be important for me to ever consciously consider the value of my all-things-considered p(doom), so long as it's high enough that I find it unacceptable to just let other people handle it. Over time, I have found myself motivated by the dire situation, and I have found myself motivated by rays of hope, but neither change what I ought to do. So why think about it?
That said, I am merely human. Worse yet, I'm a forecaster. I feel compelled to track that target. Watching my p(doom) go from 70% to 80% m...
I'm surprised to hear that there are (apparently?) many people exchanging their subjective probabilities of AI doom without that just being the top-level intro to a much more involved discussion. I feel like it's obvious that p(doom) is a flattening of all of one's thinking on AI risk into a single metric, and all the interesting differences of viewpoint are in the details, so of course I wouldn't try to compare numbers in a vacuum. Most people won't answer without first providing at least a timeline and a couple conditionals anyway.
P(doom) might be net negative as a meme -- I'm not sure, so I generally don't use that specific phrase these days, especially with people who don't already know it.
We've gotten plenty of left turns, whether sharp or no. All you have to do is make a more powerful version of what we have today, and we already know it would betray us in important ways. I think it's more likely that a new paradigm will emerge, and who's to say what the safety properties of that will be, other than "probably not what we want them to be by default," especially if it's part of an uncontrolled autonomous R&D loop and/or operating outside of human oversight.
I've always admired how well you use explicit models. In the course of my forecasting, I've found that I have to deeply trust my intuition (after vacuuming up all the relevant information I can, hiding nothing from myself). When my intuition fails, I don't double down, but I also don't wave the white flag and cast my intuition aside. I sit with the error until I feel like my intuition has updated. Then I make that mistake just a little bit less, having fed the black box that knows things I know not how.
Explicit modeling is one of many available starting po...
Not so long as more capability means more usefulness to industrial-scale clients. A few months' lead is worth a significant chunk of change. The full versions of the models will be served only to whitelisted clients, which also reduces distillability.
The capability floor is also important, not just the ceiling. If the floor continues to be raised, a lot of people will want to use the models that "don't make mistakes anymore".
This doesn't work if cognitive ability itself is the dangerous part. AI killing us via macroscale manufacturing is a plausible floor for the strangeness of doom, not the ceiling. In reality, if a superintelligence can communicate with anyone, that on its own is exceedingly dangerous. If it gains internet access, it already sees "checkmate in 4058," so to speak. It would need to be aligned before that point, which probably means it would need to be aligned before it finally gets good at philosophy.