The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Sorry, that was awfully vague. I'm probably referring to younger kids than you are. Although there's also going to be a lot of variance in when kids develop different introspective skills and conceptual understanding of their own minds.

I agree that not knowing you have a mind doesn't prevent you from having an experience. What I meant was that at some young age, I'm thinking up to age five but even 5-year-olds could be more advanced, a kid will not be able to conceptualize that they are a mind or alternately phrased that they have a mind. Nonetheless, they are a mind that is doing a bunch of complex processing and remembering some of it.

I definitely agree that phenomenal consciousness and intelligence are different. Discussions of consciousness usually break down in difficulties with terminology and how to communicate about different aspects of consciousness. I haven't come up with good ways to talk about this stuff.

I wouldn't make that argument. I just don't see the point of keeping it real.

It just seems like going virtual opens up a lot of possbilities with no downside. If you want consistency and real work, put it in your sim. Share it with other real people if you want to compromise on what worlds you'll inhabit and what challenges you'll face.

If you want real people who are really suffering so there are real stakes to play for, well, that's an orthogonal issue. I'd rather see nonconsensual suffering eliminated.

So: Why prefer the Real? What's it got that the Virtual doesn't?

Trying to align LLMs just doesn't seem optional to me. It's what's happening whether or not we like it.

Agreed on all points. Including that you writing a more detailed version more for the prosaic crowd isn't probably the best next step. That's what I was trying to do in LAMRAG, and it was no more successful than this. That's despite me starting much closer to the standard prosaic alignment/LLM-based model of AGI internals.

I think one place this argument may break down for people is the metaphor of building for the ocean as a difficult project. Maybe the lake is a lot like the ocean, and ocean storms just aren't that bad, so you can just build it to double the strength it would need on the lake and you're good to go.

My vague take on this is that what we're doing in training now is a far cry from what a nascent AGI will see in deployment, so the metaphor holds. I wonder if some well-considered optimists are assuming we'll dramatically improve training, including thinking a lot harder about what AGIs will face in deployment, before they're deployed.

If I was confident developers would at least try to do that, I'd be a good bit more optimistic.

Just some thoughts.

More thoughts have arisen. Whatever the next step is, I think this work of clarifying and specifying the arguments, could be critical. I don't think there's much chance that development will slow down let alone stop based on arguments, unless we produce far better arguments that alignment is hard and likely to fail. The abstract arguments here can just be countered by equally abstract arguments that alignment is possible because humans have it, and hey Claude seems to be doing pretty well, so a future better version of it should be fine. That apparent equality of arguments allows motivated reasoning to play tiebreaker. And more people are currently motivated toward than away from AGI.

OTOH I do think development might be deployed when the public gets involved. They have less motivated reasoning toward assuming alignment is easy, so the obvious intuition "if nobody knows how dangerous it is, we should stop!" combined with "ummm I'd like to not be permanently jobless" might make a powerful political movement for slowing/stopping. But that's an entirely separate project from arguing the case for alignment difficulty on its merits. And I don't know the first thing about public relations/marketing.

If you can't tell the difference, how could you care which is which?

I'm talking about blocking your memories of living in a simulated world.

Consciousness is very unlikely to be a binary property. Most things aren't. But there appears to be a very strong tendency for even rationalists to make this assumption in how they frame and discuss the issue.

The same is probably true of moral worth.

Taken this way, LLMs (and everything else) are partly conscious and partly deserves moral consideration. What you consider consciousness and what you consider morally worthy are to some degrees matters of opinion, but they very much depend on facts about the minds involved, so there are routes forward.

IMO current LLMs probably have a small amount of what we usually call phenomenal consciousness or qualia. They have rich internal representations and can introspect and reflect on them. But neither is nearly as rich as in a human, particularly an adult human who's learned a lot of introspection skills (including how to "play back" and interrogate contents of global workspace). Kids don't even know they have minds, let alone what's going on in there; figuring out how to figure that out is quite a learning process.

What people usually mean by "consciousness" seems to be "what it's like to be a human" which involves everything about brain function, focusing particularly on introspection - what we can directly tell about human brain function. But human consciousness is just one point on a multidimensional spectrum of different mind-properties including types of introspection.

I would question anyone who's nice to LLMs but eats factory-farmed meat. Skilled use of language is a really weird and somewhat self-centered criteria for moral worth.

Anyway, I'm also nice to LLMs because why not, and I think they probably appreciate it a tiny bit.

Future versions will have a lot more consciousness by various definitions. That's when these discussions will become widespread and perhaps even make some progress, at least in select circles like these.

One big payoff is the effect on AI safety. I expect anti-AI-slavery movements to slow down progress somewhat, maybe a lot if we don't undercut them (because motivated reasoning from not wanting to lose your job will pull people toward "oh yeah obviously they're conscious and shouldn't be enslaved!" even if the reality is that they're barely conscious. On the other hand, "free the AI" movements could be dangerous if we're trying to control maybe-misaligned TCAI.

Honesty about their likely status would make a pleasant tiebreaker in this dilemma.

This has a lot of overlap with my recent post LLM AGI may reason about its goals and discover misalignments by default, and the followup post I'm working on now that further explores whether we should train LLMs on reasoning about their goals. Prompting them to reason extensively about goals during training has the effect of revealing potential future misalignments to them, as you discuss. 

I'm curious what you think about that framing.

Even humans have taken over the world. Something a little smarter should have a fairly easy time.

I do agree that there's a soft limit above human level for LLMs/agents, but it's not a hard limit and it's not right at human level.

I think this sometimes has a good explanation in the 80/20 rule. Which itself is based on a pretty deep tendency for things to be complex. So changing things often has some low-hanging fruit (in the idealized case, the 80% you can get for 20% of the effort). It's easy to eliminate some clutter, harder to eliminate the rest.

This isn't always the case. In many of your examples, there's no obvious reason that cutting to zero would be harder. In some there are.

Here's another one: instead of cutting clutter to zero or time with that friend to zero, maybe you use that time /effort to go get some low-hanging fruit in other areas of life optimization?