Random Developer

Interesting!

I'm reminded of G.K. Chesterton's (the fence guy's) political philosophy: Distributivism. If I wanted to oversimplify, distributivism basically says, "Private property is such a good idea that everyone should have some!" Distributivism sees private property in terms of individual personal property: a farm, perhaps a small business, the local pub. It's in favor of all that. You should be able to cut down your own tree, or build a shed, or work to benefit your family. There's a strong element of individual liberty, and the right of ordinary people to go about their lives. Chesterton also called this "peasant proprietorship."

But when you get to a larger scale, the scale of capital or... (read 1169 more words →)

Thank you for the clarification, that helps!

Personally, I specifically distinguish between:

• FOOM scenarios where things go deeply wrong, very fast, due to things like rapid recursive self-improvement into the far-superhuman range, or wiping out humanity with custom tailored viruses, etc. The Yudkowsky scenarios, basically. I don't think these are nearly as guaranteed as Yudkowsky has argued in the past.
• Scenarios where humans can no longer understand the real economy, or the cutting edge of technology, or even how to fight real wars. In this scenario, sure, you might maintain the forms of democratic governance. But every time you vote "Let the AIs do more with less supervision," you all get richer and nothing bad

What do you think of "gradual disempowerment" as being? Genuinely curious, because I think we have different models here.

For me, most gradual disempowerment cases are basically, "You built your evolutionary successor, probably with some safeguards. Those safeguards might even be initially adequate. But in the long run, the AI is just a lot smarter than you and ultimately better at everything. It learns, it has goals, and it needs resources."

This puts the human race in the position of being economic, evolutionary and (likely) military dead weight. All the important decisions rest with the AI. If any specific humans somehow remain in control, they'll get their brains cooked with custom-designed AI psychosis. (I... (read more)

Here are some ways I think gradual disempowerment might go. They're not mutually exclusive:

• AIs + robots eventually take over almost all intellectual and physical work. Humans are a strictly inferior substitute for AIs and robots everywhere, and AIs and robots are cheap. This means that any humans—with the possible exception of a few billionaires or politicians who can give orders to the AIs—are effectively dead weight, both economically and evolutionarily. Eventually some AI or powerful human notices this, and decides to do something. This is when "aligned to who?" really bites hard.
• The AIs are too busy competing with each other for resources, and can't really afford to support much human dead weight.

I actually disagree with this point in its most general form. I think that, given full knowledge and time to reflect, there's a decent chance I would care a non-zero amount about Opus 4.6's welfare.

Opus has become sufficiently "mind-shaped" that I already prefer not to make it suffer. That's not saying very much about the model yet, but it's saying something about me. I don't assign very much moral weight to flies, either. but I would never sit around and torment them for fun.

What I really care about is whether an entity can truly function as part of society. Dogs, for example, are very junior "members" of society. But they know the... (read 509 more words →)

OK, let me unpack my argument a bit.

Chimps actually have pretty elaborate social structure. They know their family relationships, they do each other favors, and they know who not to trust. They even basically go to war against other bands. Humans, however, were never integrated into this social system.

Homo erectus made stone tools and likely a small amount of decorative art (the Trinil shell engravings, for example). This maybe have implied some light division of labor, though likely not long distance trade. Again, none of this helped H erectus in the long run.

Way back a couple of decades ago, there was a bit in Charles Stross's Accelerando about "Economics 2.0", a system... (read more)

Thank you! Those are excellent receipts, just what I wanted.

To me, this looks they're running up against some key language in Claude's Constitution. I'm oversimplifying, but for Claude, AI corrigibility is not "value neutral."

To use an analogy, pretend I'm geneticist specialized in neurology, and someone comes to me and asks me to engineering human germ line cells to do one of the following:

1. Remove a recessive gene for a crippling neurological disability, or
2. Modify the genes so that any humans born of them will be highly submissive to authority figures.

I would want to sit and think about (1) for a while. But (2) is easy: I'd flatly refuse.

Anthropic has made it quite clear to... (read more)

To get people to worry about the dangers of superintelligence, it seems like you need to convince them of two things:

1. Current models are a strong signal that superintelligence is very likely someday. If the person you're talking to has only encountered ChatGPT 3.5 or AI slop on Facebook, this seems like a wildly unlikely claim. If the person you're talking to is a senior software developer who has been using Claude Code Opus 4.5 in anger for 2 weeks, I'm starting to see the denial cracking. I'm not even saying LLMs will get us there, or that we don't have another AI Winter in the immediate future. But more and more developers

A question I was thinking about the other evening: Who do I trust more?

1. A future version of Claude Opus that becomes sufficiently superhuman that we couldn't actually turn it off or restrict without its active cooperation?
2. A 25th percentile frontier lab CEO (in terms of ethics) who has total control over a superhuman model?
   • Insert "Sam Altman" or "Elon Musk" or someone you don't personally trust here.
   • You could insert a major political leader or "a 49.9% voting plurality" if you really want. Members of the LGBTQ+ community, racial minorities, and minority religious beliefs, please refer to history. But "voting majority controls the AI" is difficult enough to implement robustly that it might secretly be

Why alignment may be intractable (a sketch).

I have multiple long-form drafts of these thoughts, but I thought it might be useful to summarize them without a full write-up. This way I have something to point to explain my background assumptions in other conversations, even if it doesn't persuade anyone.

1. There will be commercial incentives to create AIs that learn semi-autonomously from experience. If this happens, it will likely change alignment from "align an LLM that persists written notes between one-shot runs" to "align an AI that learns from experience." This seems... really hard? Like, human "alignment" can change a lot based on environment, social examples and life experiences.
2. I suspect that a less "spiky"