Your argument with Alexandros was what inspired this post, actually. I was thinking about whether or not to send this to you directly... guess that wasn't necessary.
The question is not whether I can pass their ITT: that particular claim doesn't obviously engage with any cruxes that I or others like me to have, related to x-risk. That's the only thing that section is describing.
Yeah, that seems like a plausible contributor to that effect.
Edit: though I think this is true even if you ignore "who's calling for regulations" and just look at the relative optimism of various actors in the space, grouped by their politics.
I was going to write, "surely the relevant figure is how much you pay per month, as a percentage of your income", but then I looked at the actual image and it seems like that's what you meant by house price.
Yes, right tails for things that better represent actual value produced in the world, i.e. projects/products/etc. I'm pretty skeptical of productivity metrics for individual developers like the ones described in that paper, since almost by construction they're incapable of capturing right-tail outcomes, and also fail to capture things like "developer is actually negative value". I'm open to the idea that remote work has better median performance characteristics, though expect this to be pretty noisy.
On priors I think you should strongly expect in-person co-working to produce much fatter right-tails. Communication bandwidth is much higher, and that's the primary bottleneck for generating right-tail outcomes.
I don't know how I'd evaluate that without specific examples. But in general, if you think price signals are wrong or "more misleading than not" when it comes to measuring endpoints we actually care about, then I suppose it's coherent to argue that we should ignore price signals.
Because there's a big difference between "has unsavory political stances" and "will actively and successfully optimize for turning the US into a fascist dictatorship", such that "far right or fascist" is very misleading as a descriptor.
I might agree with a more limited claim like "most people in our reference class underestimate the chances of western democracies turning into fascist dictatorships over the next decade".
I don't think someone reading this post should have >50% odds on >50% of western democracies turning into fascist dictatorships over the next decade or two, no. I don't see an argument that "fascist dictatorship" is a stable attractor; as others have pointed out, even countries which started out much closer to that endpoint have mostly not ended up there after a couple of decades despite appearing to move in that direction.
Oh, that might just be me having admin permissions, whoops. I'll double-check what the intended behavior is.
(Update: I just merged a PR that should fix the issue, i.e. make it clear who's comments got deleted. Should be live in about 7 minutes)
I don't actually see very much of an argument presented for the extremely strong headline claim:
This post aims to show that, over the next decade, it is quite likely that most democratic Western countries will become fascist dictatorships - this is not a tail risk, but the most likely overall outcome.
You draw an analogy between the "by induction"/"line go up" AI risk argument, and the increase in far-right political representation in Western democracies over the last couple decades. But the "by induction"/"line go up" argument for AI risk is not the ...
This seems like a strange reaction. If an alien read this post and believed the claims, wouldn't they think fascism was pretty likely very much on the rise? There's global trends, and there's a bunch of specific examples. Do you agree with that?
Maybe you have some reasons that this prima facie evidence isn't actually strong evidence. What are those reasons?
...But the "by induction"/"line go up" argument for AI risk is not the reason one should be worried; one should be worried for specific causal reasons that we expect unaligned ASI to cause extremely bad o
We've thought about something like a "notes to self" feature but don't have anything immediate planned. In the meantime I'd recommend a 3rd-party solution if bookmarks without notes don't do the thing you need; I've used Evernote in the past but I'm sure there are more options.
Robotic supply chain automation only seems necessary in worlds where it's either surprisingly difficult to get AGI to a sufficiently superhuman level of cognitive ability (such that it can find a much faster route to takeover), worlds where faster/more reliable routes to takeover either don't exist or are inaccessible even to moderately superhuman AGI, or some combination of the two.
At a guess (not having voted on it myself): because most of the model doesn't engage with the parts of the question that those voting consider interesting/relevant, such as the many requirements laid out for "transformative AI" which don't see at all necessary for x-risk. While this does seem to be targeting OpenPhil's given definition of AGI, they do say in a footnote:
What we’re actually interested in is the potential existential threat posed by advanced AI systems.
While some people do have AI x-risk models that route through ~full automation (or su...
You're welcome to host images wherever you like - we automatically mirror all embedded images on Cloudinary, and replace the URLS in the associated image tags when serving the post/comment (though the original image URLs remain in the canonical post/comment for you, if you go to edit it, or something).
Like, I cannot buy chicken for $0.87/lb, I pay about $6.50/lb
I'm sorry, what? Like, I can in fact go buy boneless chicken thighs for $6.50/lb at Whole Foods in the Bay Area, but that is not what the average consumer is paying. Prices are in fact more like $1/lb for drumsticks, $1.5/lb for whole birds, $3/lb for boneless thighs/breasts.
Humans have all the resources, they don’t need internet, computers, or electricity to live or wage war, and are willing to resort to extremely drastic measures when facing a serious threat.
Current human society definitely relies in substantial part on all of the above to function. I agree that we wouldn't all die if we lost electricity tomorrow (for an extended period of time), but losing a double-digit % of the population seems plausible.
Also, observably, we, as a society, do not resort to sensible measures when dealing with a serious thread (e.g. c...
Yeah, I probably should have explicitly clarified that I wasn't going to be citing my sources there. I agree that the fact that it's costly to do so is a real problem, but Robert Miles points out, some of the difficulty here is insoluble.
It's very strange to me that there isn't a central, accessible "101" version of the argument given how much has been written.
There are several, in fact; but as I mentioned above, none of them will cover all the bases for all possible audiences (and the last one isn't exactly short, either). Off the top of of my...
the presentation of the topic is unpersuasive to an intelligent layperson
There is, of course, no single presentation, but many presentations given by many people, targeting many different audiences. Could some of those presentations be improved? No doubt.
I agree that the question of how to communicate the problem effectively is difficult and largely unsolved. I disagree with some of the specific prescriptions (i.e. the call to falsely claim more-modest beliefs to make them more palatable for a certain audience), and the object-level argum...
Over the years roughly between 2015 and 2020 (though I might be off by a year or two), it seemed to me like numerous AI safety advocates were incredibly rude to LeCun, both online and in private communications.
I'd be interested to see some representative (or, alternatively, egregious) examples of public communications along those lines. I agree that such behavior is bad (and also counterproductive).
Against them, The conjecture about what protein folding and ribosomes might one have the possibility to do really weak counterargument, based as it is on no empirical or evidentiary reasoning
I'm not sure I've parsed this correctly, but if I have, can I ask what unsupported conjecture you think undergirds this part of the argument? It's difficult to say what counts as "empirical" or "evidentiary" reasoning in domains where the entire threat model is "powerful stuff we haven't managed to build ourselves yet", given we can be confident that set isn't em...
By contrast, some lines of research where I’ve seen compelling critiques (and haven’t seen compelling defences) of their core intuitions, and therefore don't recommend to people:
- Cooperative inverse reinforcement learning (the direction that Stuart Russell defends in his book Human Compatible); critiques here and here.
- John Wentworth’s work on natural abstractions; exposition and critique here, and another here.
The first critique of natural abstractions says:
...Concluding thoughts on relevance to alignment: While we’ve made critical rem
Some combination of:
There seems to be much more diversity in human cognitive performance than there is in human-brain-energy-efficiency; whether this is due to larger differences in the underlying software (to the extent that this is meaningfully commensurable with differences in hardware) or because smaller differences in that domain result in much larger differences in observable outputs, or both, none of that really takes away from the fact that brain software does not seem to be anywhere near the relevant efficiency frontier, especially since many trade-offs which were operative at an evolutionary scale simply aren't when it comes to software.
I've heard people be somewhat optimistic about this AI guideline from China. They think that this means Beijing is willing to participate in an AI disarmament treaty due to concerns over AI risk.
I'm curious where you've seen this. My impression from reading the takes of people working on the governance side of things is that this is mostly being interpreted as a positive sign because it (hopefully) relaxes race dynamics in the US. "Oh, look, we don't even need to try all that hard, no need to rush to the finish line." I haven't seen anyone seri...
I've only seen vaguely positive vibes from people who needed Google Translate in order to understand it, like this post from Zvi. He comes to the conclusion that "All of that points, once again, to an eager partner in a negotiation." This isn't obvious at all. Again, a willingness to regulate nuclear power is not a strong signal for a willingness to participate in nuclear disarmament treaty - all states will eventually have some level of nuclear regulation.
which implies by association that brain software is much more efficient as it was produced by exactly the same evolutionary process which he now admits produced fully optimized conventional computational elements over the same time frame, etc
I don't believe this would follow; we actually have much stronger evidence that ought to screen off that sort of prior - simply the relatively large differences in human cognitive abilities.
My guess is the interpretability team is under a lot of pressure to produce insights that would help the rest of the org with capabilities work
I would be somewhat surprised if this was true, assuming you mean a strong form of this claim (i.e. operationalizing "help with capabilities work" as relying predominantly on 1st-order effects of technical insights, rather than something like "help with capabilities work by making it easier to recruit people", and "pressure" as something like top-down prioritization of research directions, or setting KPIs which rely...
Complexity of value is part of why value is fragile.
(Separately, I don't think current human efforts to "figure out" human values have been anywhere near adequate, though I think this is mostly a function of philosophy being what it is. People with better epistemology seem to make wildly more progress in figuring out human values compared to their contemporaries.)
That does indeed seem like some progress, though note that it does not really let us answer questions like "what algorithm is this NN performing that lets it do whatever it's doing", to a degree of understanding sufficient to implement that algorithm directly (or even a simpler, approximated version, which is still meaningfully better than what the previous state-of-the-art was, if restricted to "hand-written code" rather than an ML model).
I am not saying that. Many libertarians think that centralization of power often has bad effects. But trying to argue with libertarians who are advocating for government regulations because they're worried about AI x-risk by pointing out that government regulation will increase centralization of power w.r.t. AI is a non-sequitur, unless you do a lot more work to demonstrate how the increased centralization of power acts contrariwise the libertarian's goals in this case.