Posts
Wiki Contributions
Comments
I think Six Dimensions of Operational Adequacy was in this direction; I wish we had been more willing to, like, issue scorecards earlier (like publishing that document in 2017 instead of 2022). The most recent scorecard-ish thing was commentary on the AI Safety Summit responses.
I also have the sense that the time to talk about unpausing is while creating the pause; this is why I generally am in favor of things like RSPs and RDPs. (I think others think that this is a bit premature / too easy to capture, and we are more likely to get a real pause by targeting a halt.)
While the coauthors broadly agree about points listed in the post, I wanted to stick my neck out a bit more and assign some numbers to one of the core points. I think on present margins, voluntary restraint slows down capabilities progress by at most 5% while probably halving safety progress, and this doesn't seem like a good trade. [The numbers seem like they were different in the past, but the counterfactuals here are hard to estimate.] I think if you measure by the number of people involved, the effect of restraint is substantially lower; here I'm assuming that people who are most interested in AI safety are probably most focused on the sorts of research directions that I think could be transformative, and so have an outsized impact.
Similarly for the Sierra Club, I think their transition from an anti-immigration org to a pro-immigration org seems like an interesting political turning point that could have failed to happen in another timeline.
From the outside, Finnish environmentalism seems unusually good--my first check for this is whether or not environmentalist groups are pro-nuclear, since (until recently) it was a good check for numeracy.
Note that the 'conservation' sorts of environmentalism are less partisan in the US, or at least, are becoming partisan later. (Here's an article in 2016 about a recent change of a handful of Republicans opposed to national parks, in the face of bipartisan popular support for them.) I think the thing where climate change is a global problem instead of a local problem, and a conflict between academia and the oil industry, make it particularly prone to partisanship in the US. [Norway also has significant oil revenues--how partisan is their environmentalism, and do they have a similar detachment between conservation and climate change concerns?]
I think this is true of an environmentalist movement that wants there to be a healthy environment for humans; I'm not sure this is true of an environmentalist movement whose main goal is to dismantle capitalism. I don't have a great sense of how this has changed over time (maybe the motivations for environmentalism are basically constant, and so it can't explain the changes), but this feels like an important element of managing to maintain alliances with politicians in both parties.
(Thinking about the specifics, I think the world where Al Gore became a Republican (he was a moderate for much of his career) or simply wasn't Clinton's running mate (which he did in part because of HW Bush's climate policies) maybe leads to less partisanship. I think that requires asking why those things happened, and whether there was any reasonable way for them to go the other way. The oil-republican link seems quite strong during the relevant timeframe, and you either need to have a strong oil-democrat link or somehow have a stronger climate-republican link, both of which seem hard.)
I get that this is the first post out of 4, and I'm skimming the report to see if you address this, but it sounds like you're using historical data to try to prove a counterfactual claim. What alternative do you think was possible? (I assume the presence of realistic alternatives is what you mean by 'not inevitable', but maybe you mean something else.)
I think the main feature of AI transition that people around here missed / didn't adequately foreground is that AI will be worse is better. AI art will be clearly worse than the best human art--maybe even median human art--but will cost pennies on the dollar, and so we will end up with more, worse art everywhere. (It's like machine-made t-shirts compared to tailored clothes.) AI-enabled surveillance systems will likely look more like shallow understanding of all communication than a single overmind thinking hard about which humans are up to what trouble.
This was even hinted at by talking about human intelligence; this comment is from 2020, but I remember seeing this meme on LW much earlier:
When you think about it, because of the way evolution works, humans are probably hovering right around the bare-minimal level of rationality and intelligence needed to build and sustain civilization. Otherwise, civilization would have happened earlier, to our hominid ancestors.
Similarly, we should expect widespread AI integration at about the bare-minimum level of competence and profitability.
I often think of the MIRI view as focusing on the last AI; I.J. Good's "last invention that man need ever make." It seems quite plausible that those will be smarter than the smartest humans, but possibly in a way that we consider very boring. (The smartest calculators are smarter than the smartest humans at arithmetic.) Good uses the idea of ultraintelligence for its logical properties (it fits nicely into a syllogism) rather than its plausibility.
[Thinking about the last AI seems important because choices we make now will determine what state we're in when we build the last AI, and aligning it is likely categorically different from aligning AI up to that point, so we need to get started now and try to develop in the right directions.]
1A lot of this depends on where you draw the line between 'rationality' and 'science' or 'economics' and 'philosophy' or so on. As well, given that 'rationality' is doing the best you can given the constraints you're under, it seems likely that many historical figures were 'rational' even if they weren't clear precursors to the modern rationalist cluster.
For example, I think Xunzi (~3rd century BCE) definitely counts; check out Undoing Fixation in particular. [His students Li Si and Han Fei are also interesting in this regard, but I haven't found something by them yet that makes them clearly stand out as rationalists. Also, like JenniferRM points out, they had a troubled legacy somewhat similar to Alexander's.]
Some people count Mozi as the 'first effective altruist' in a way that seems similar.
People point to Francis Bacon as the originator of empiricism; you can read his main work here on LW. While influential in English-language thought, I think he is anticipated by al-Haytham and Ibn Sina.
LaPlace is primarily famous as a mathematician and scientist, but I think he was important in the development of math underpinning modern rationality, and likely counts.
Benjamin Franklin seems relevant in a handful of ways; his autobiography is probably the best place to start reading.
Alfred Korzybski is almost exactly a hundred years older than Yudkowsky, and is the closest I'm aware of to rationality-as-it-is-now. You can see a discussion of sources between then and now in Rationalism Before The Sequences.
1What would be a better framing?
I talk about something related in self and no-self; the outward-flowing 'attempt to control' and the inward-flowing 'attempt to perceive' are simultaneously in conflict (something being still makes it easier to see where it is, but also makes it harder to move it to where it should be) and mutually reinforcing (being able to tell where something is makes it easier to move it precisely where it needs to be).
Similarly, you can make an argument that control without understanding is impossible, that getting AI systems to do what we want is one task instead of two. I think I agree the "two progress bars" frame is incorrect but I think the typical AGI developer at a lab is not grappling with the philosophical problems behind alignment difficulties, and is trying to make something that 'works at all' instead of 'works understandably' in the sort of way that would actually lead to understanding which would enable control.
I... wish people wouldn't do this? Or, like, maybe you should ask Claude for the search terms to use, but going to a grounded source seems pretty important to staying grounded.