Recent generations of Claude seem better at understanding and making fairly subtle judgment calls than most smart humans. These days when I’d read an article that presumably sounds reasonable to most people but has what seems to me to be a glaring conceptual mistake, I can put it in Claude, ask it to identify the mistake, and more likely than not Claude would land on the same mistake as the one I identified.
I think before Opus 4 this was essentially impossible, Claude 3.xs can sometimes identify small errors but it’s a crapshoot on whether it can identify central mistakes, and certainly not judge it well.
It’s possible I’m wrong about the mistakes here and Claude’s just being sycophantic and identifying which things I’d regard as the central mistake, but if that’s true in some ways it’s even more impressive.
Interestingly, both Gemini and ChatGPT failed at these tasks.
For clarity purposes, here are 3 articles I recently asked Claude to reassess (Claude got the central error in 2/3 of them). I'm also a little curious what the LW baseline is here; I did not include my comments in my prompts to Claude.
https://terrancraft.com/2021/03/21/zvx-the-effects-of-scouting-pillars/
https://www.clearerthinking.org/post/what-can-a-single-data-point-teach-you
https://www.lesswrong.com/posts/vZcXAc6txvJDanQ4F/the-median-researcher-problem-1
It's all relative. I think my article is more moderate than say some of the intros on LW but more aggressive/direct than say Kelsey Piper's or Daniel Eth's intros, and maybe on par with dynomight's (while approaching it from a very different angle).
You might also like those articles more; ironically from my perspective I deliberately crafted this article to drop many signifiers of uncertainty that is more common in my natural speech/writing, on the theory that it's more pleasant/enjoyable to read.
The graphs are a bit difficult to read, but see figure 2 here. Note that this is the writeup of the 2023 Grace et.al survey that also includes answers from the 2022 survey (In 2023 there's a large jump backwards in timelines across the board, which is to be expected).
I didn't include the source in the main article because I was worried it'd be annoying to parse, and explaining this is hard (not very hard, just that it requires more than one sentence and a link and I was trying to be economic in my opening).
I like Scott's Mistake Theory vs Conflict Theory framing, but I don't think this is a complete model of disagreements about policy, nor do I think the complete models of disagreement will look like more advanced versions of Mistake Theory + Conflict Theory.
To recap, here's my short summaries of the two theories:
Mistake Theory: I disagree with you because one or both of us are wrong about what we want, or how to achieve what we want)
Conflict Theory: I disagree with you because ultimately I want different things from you. The Marxists, who Scott was originally arguing against, will natively see this as about individual or class material interests but this can be smoothly updated to include values and ideological conflict as well.
I polled several rationalist-y people about alternative models for political disagreement at the same level of abstraction of Conflict vs Mistake, and people usually got to "some combination of mistakes and conflicts." To that obvious model, I want to add two other theories (this list is incomplete).
First, consider Thomas Schelling's 1960 opening to Strategy of Conflict
The book has had a good reception, and many have cheered me by telling me they liked it or learned from it. But the response that warms me most after twenty years is the late John Strachey’s. John Strachey, whose books I had read in college, had been an outstanding Marxist economist in the 1930s. After the war he had been defense minister in Britain’s Labor Government. Some of us at Harvard’s Center for International Affairs invited him to visit because he was writing a book on disarmament and arms control. When he called on me he exclaimed how much this book had done for his thinking, and as he talked with enthusiasm I tried to guess which of my sophisticated ideas in which chapters had made so much difference to him. It turned out it wasn’t any particular idea in any particular chapter. Until he read this book, he had simply not comprehended that an inherently non-zero-sum conflict could exist. He had known that conflict could coexist with common interest but had thought, or taken for granted, that they were essentially separable, not aspects of an integral structure. A scholar concerned with monopoly capitalism and class struggle, nuclear strategy and alliance politics, working late in his career on arms control and peacemaking, had tumbled, in reading my book, to an idea so rudimentary that I hadn’t even known it wasn’t obvious.
I claim that this "rudimentary/obvious idea," that the conflict/cooperative elements of many human disagreements is structurally inseparable, is central to a secret third thing distinct from Conflict vs Mistake[1]. If you grok the "obvious idea," we can derive something like
Negotiation Theory(?): I have my desires. You have yours. I sometimes want to cooperate with you, and sometimes not. I take actions maximally good for my goals and respect you well enough to assume that you will do the same; however in practice a "hot war" is unlikely to be in either of our best interests.
In the Negotiation Theory framing, disagreement/conflict arises from dividing the goods in non-zerosum games. I think the economists/game theorists' "standard models" of negotiation theory is natively closer to "conflict theory" than "mistake theory." (eg, often their models assume rationality, which means the "can't agree to disagree" theorems apply). So disagreements are due to different interests, rather than different knowledge. But unlike Marxist/naive conflict theory, we see that conflicts are far from desired or inevitable, and usually there are better trade deals from both parties' lights than not coordinating, or war.
(Failures from Negotiation Theory's perspectives often centrally look like coordination failures, though the theory is broader than that and includes not being able to make peace with adversaries)/
Another framing that is in some ways a synthesis and in some ways a different view altogether that can sit in each of the previous theories is a thing that many LW people talk about, but not exactly in this context:
Motivated Cognition: People disagree because their interests shape their beliefs. Political disagreements happen because one or both parties are mistaken about the facts, and those mistakes are downstream of material or ideological interests shading one's biases. Upton Sinclair: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Note the word "difficult," not "impossible." This is Sinclair's view and I think it's correct. Getting people to believe (true) things against their material interests to believe is possible but the skill level required is higher than a neutral presentation of the facts to a neutral third party.
Interestingly, the Motivated Cognition framing suggests that there might not be a complete truth of the matter for whether "Mistake Theory" vs "Conflict Theory" vs Negotiation Theory is more correct for a given political disagreement. Instead, your preferred framing has a viewpoint-dependent and normative element to it.
Suppose your objective is just to get a specific policy passed (no meta-level preferences like altruism), and you believe this policy is in your interests and those of many others, and people who oppose you are factually wrong.
Someone who's suited to explanations like Scott (or like me?), might naturally fall into a Mistake Theory framing, and write clear-headed blogposts about why people who disagree with you are wrong. If the Motivated Cognition theory is correct, most people are at least somewhat sincere, and at some level of sufficiently high level of simplicity, people can update to agree with you even if it's not immediately in their interests (smart people in democracies usually don't believe 2+2=5 even in situations where it'd be advantageous for them to do so)
Someone who's good at negotiations and cooperative politics might more naturally adopt a Negotiation Theory framing, and come up with a deal that gets everybody (or enough people) what they want while having their preferred policy passed.
Finally, someone who's good at (or temperamentally suited to) non-cooperative politics and the more Machiavellian side of politics might identify the people who are most likely to oppose their preferred policies, and destroy their political influence enough that the preferred policy gets passed.
Anyway, here are my four models of political disagreement (Mistake, Conflict, Negotiation, Motivated Cognition). I definitely don't think these four models (or linear combinations of them) explain all disagreements, or are the only good frames for thinking of disagreement. Excited to hear alternatives [2]!
[1] (As I was writing this shortform, I saw another, curated, post comparing nonzero sum games against mistake vs conflict theory. I thought it was a fine post but it was still too much in the mistake vs conflict framing, whereas arguably in my eyes the nonzero sum game view is an entirely different way to view the world)
[2] In particular I'm wondering if there is a distinct case for ideological/memetic theories that operate along a similar level of abstraction as the existing theories, as opposed to thinking of ideologies as primarily given us different goals (which would make them slot in well with all the existing theories except Mistake Theory).
I find it very annoying when people give dismissals of technology trendlines because they don't have any credence in straight lines on a graph. Often people will post a meme like the following, or something even dumber.
I feel like it's really obvious why the two situations are dissimilar, but just to spell it out: the growth rate of human children is something we have overwhelming evidence for. Like literally we have something like 10 billion to 100 billion data points of extremely analogous situations against the exponential model.
And this isn't even including the animal data! Which should conservatively give you another 10-100x factor difference.
(This is merely if you're a base rates/forecasting chartist like me, and without even including physical and biological plausibility arguments).
So if someone starts with an exponential model of child growth, with appropriate error bars, they quickly update against it because they have something like > A TRILLION TO ONE EVIDENCE AGAINST.
But people think their arguments against Moore's Law, or AI scaling lows, or the METR time horizon curve, or w/e, merely because they don't believe in lines on a curve are sneaking in the assumption that "it definitely doesn't work" from a situation with a trillion to one evidence against on chartist grounds, plus physical plausibility arguments, when they have not in fact provided a trillion to one evidence against in their case.
1
1
1My guess is that it's a good fit for other intros but not this one. My guess is that most readers are already attuned to the idea that "tech company CEOs having absolute control over radically powerful and transformative technologies may not be good for me", so the primary advantages of including it in my article are:
- Signaling: Giving readers assurances that I'm not naive to those risks
- Precisification: Giving readers more precise models for how exactly this could go bad whereas before they might only have loose models.
Against those advantages I'm balancing against a) making the article even longer and more confusing to navigate (this article isn't maximally long but it's like 2500 words not including footnotes and captions, and when we were conceptualizing this article in the abstract Claude and I were targeting more like 1k-1.2k words), and b) making the "bad AI CEO taking over the world" memes swamp other messages.
But again, I think this is just my own choice for this specific article. I think other people should talk about concentration-of-power risks at least sometimes, and I can imagine researching or writing more about it in the future for other articles myself too.
Fair, depending on your priors there's definitely an important sense in which something like Reardon's case is simpler:
https://frommatter.substack.com/p/the-bone-simple-case-for-ai-x-risk
I'd be interested in someone else trying to rewrite his article while removing in-group jargon and tacit assumptions!
Yeah each core point has some number of subpoints. I'm curious if you think I should instead have instantiated each point with just the strongest, or easiest to explain, subpoint (especially when it was branching). Eg the current structure looks like
1: The world’s largest tech companies are building intelligences that will become better than humans at almost all economically and militarily relevant tasks
1.1 (implicit) building intelligences that will become better than humans at almost all economically relevant tasks
1.2 (implicit) building intelligences that will become better than humans at almost all militarily relevant tasks
I can imagine a version that picks one side of the tree and just focuses on economic tasks, or military tasks.
2: Many of these intelligences will be goal-seeking minds acting in the real world, rather than just impressive pattern-matchers
2.1 goal-seeking minds ("agentic" will be ML parlance, but I was deliberately trying to avoid that jargon)
2.2 acting in the real world.
2.3 existing efforts to make things goal-seeking
2.4 the trend
2.5 selection pressure to make them this way.
This has 5 subpoints (really 6 since it's a 2 x 3 tuple).
Maybe the "true" simplest argument will just pick one branch, like selection pressures for goal-seeking minds.
And so forth.
My current guess is the true minimalist argument, at least presented at the current level of quality/given my own writing skill, will be substantially less persuasive, but this is only weakly held. I wish I had better intuitions for this type of thing!
The article's now out! Comments appreciated
https://linch.substack.com/p/simplest-case-ai-catastrophe
1
base rates, man, base rates.