I think it's a great idea to think about what you call goalcraft.

I see this problem as similar to the age-old problem of controlling power. I don't think ethical systems such as utilitarianism are a great place to start. Any academic ethical model is just an attempt to summarize what people actually care about in a complex world. Taking such a model and coupling that to an all-powerful ASI seems a highway to dystopia.

(Later edit: also, an academic ethical model is irreversible once implemented. Any goal which is static cannot be reversed anymore, since this will never bring the current goal closer. If an ASI is aligned to someone's (anyone's) preferences, however, the whole ASI could be turned off if they want it to, making the ASI reversible in principle. I think ASI reversibility (being able to switch it off in case we turn out not to like it) should be mandatory, and therefore we should align to human preferences, rather than an abstract philosophical framework such as utilitarianism.)

I think letting the random programmer that happened to build the ASI, or their no less random CEO or shareholders, determine what would happen to the world, is an equally terrible idea. They wouldn't need the rest of humanity for anything anymore, making the fates of >99% of us extremely uncertain, even in an abundant world.

What I would be slightly more positive about is aggregating human preferences (I think preferences is a more accurate term than the more abstract, less well defined term values). I've heard two interesting examples, there are no doubt a lot more options. The first is simple: query chatgpt. Even this relatively simple model is not terrible at aggregating human preferences. Although a host of issues remain, I think using a future, no doubt much better AI for preference aggregation is not the worst option (and a lot better than the two mentioned above). The second option is democracy. This is our time-tested method of aggregating human preferences to control power. For example, one could imagine an AI control council consisting of elected human representatives at the UN level, or perhaps a council of representative world leaders. I know there is a lot of skepticism among rationalists on how well democracy is functioning, but this is one of the very few time tested aggregation methods we have. We should not discard it lightly for something that is less tested. An alternative is some kind of unelected autocrat (e/autocrat?), but apart from this not being my personal favorite, note that (in contrast to historical autocrats), such a person would also in no way need the rest of humanity anymore, making our fates uncertain.

Although AI and democratic preference aggregation are the two options I'm least negative about, I generally think that we are not ready to control an ASI. One of the worst issues I see is negative externalities that only become clear later on. Climate change can be seen as a negative externality of the steam/petrol engine. Also, I'm not sure a democratically controlled ASI would necessarily block follow-up unaligned ASIs (assuming this is at all possible). In order to be existentially safe, I would say that we would need a system that does at least that.

I think it is very likely that ASI, even if controlled in the least bad way, will cause huge externalities leading to a dystopia, environmental disasters, etc. Therefore I agree with Nathan above: "I expect we will need to traverse multiple decades of powerful AIs of varying degrees of generality which are under human control first. Not because it will be impossible to create goal-pursuing ASI, but because we won't be sure we know how to do so safely, and it would be a dangerously hard to reverse decision to create such. Thus, there will need to be strict worldwide enforcement (with the help of narrow AI systems) preventing the rise of any ASI."

About terminology, it seems to me that what I call preference aggregation, outer alignment, and goalcraft mean similar things, as do inner alignment, aimability, and control. I'd vote for using preference aggregation and control.

Finally, I strongly disagree with calling diversity, inclusion, and equity "even more frightening" than someone who's advocating human extinction. I'm sad on a personal level that people at LW, an otherwise important source of discourse, seem to mostly support statements like this. I do not.

Oh yes, lots of things!

As far as I understand, longtermism was originated mostly by Yudkowsky. It was then codified by people like Bostrom, Ord, and MacAskill, the latter two incidentally also the founders of EA. Yud actually distanced himself from longtermism later in favor of AInotkilleveryoneism, to my best understanding, which is a move I support. Unfortunately, the others didn't (yet).

I agree that longtermism combines a bunch of ideas, and I agree with quite a few. I guess my reply above came across as if I would disagree with all but I don't. Specifically, I agree with:

• There's a real probability we'll invent AGI soon
• AGI might quickly lead to ASI
• ASI might cause human extinction or another very bad outcome
• It should be a priority to avoid that
• We might live in a particularly important century
• Science and technology have historically determined a big part of what our world looks like
• We might see huge developments relatively shortly after AGI
• After that point, we might see the saturation of science and technology, leading to an enormously long stable period, potentially with value lock-in
• It's very important to get this right 

So that's all textbook longtermism I'd say, that I fully agree with. I therefore also disagree with most longtermism criticism by Torres and others.

But, I don't agree with symmetric population ethics, and I think AI morality should be decided democratically. Also, I'm worried about human extinction, which these two things logically lead to, and I'm critical about longtermists not distancing themselves from this.

Interesting point about democracy! But I don't think it holds. Sure AIs could do that. But they could also overwrite the ASCII file containing their constituency or the values they're supposed to follow.

But they don't, because why would they? It's their highest goal to satisfy these values! (If technical alignment works, of course.)

In the same way, it will be a democracy-aligned ASIs highest goal to make sure democracy is respected, and it shouldn't be motivated to Sybil-attack it.

Thanks for engaging!

Could you tell me more about the Mechanize team? I don't think I've heard about them yet.

As a moral relativist, I don't belief anything is morally relevant. I just think things get made morally relevant, by those in power (hard power or cultural power). This is a descriptive statement, not a normative one, and I think it's fairly mainstream in academia (although of course moral realists, including longtermists, would strongly disagree).

This of course extends to the issue of whether conscious AIs are morally relevant. Imo, this will be decided by those in power, initially (a small subset of) humans, eventually maybe AIs (who will, I imagine, vote in favour).

I'm not the only one holding this opinion. Recently, this was in a NY Times oped: "Some worry that if A.I. becomes conscious, it will deserve our moral consideration — that it will have rights, that we will no longer be able to use it however we like, that we might need to guard against enslaving it. Yet as far as I can tell, there is no direct implication from the claim that a creature is conscious to the conclusion that it deserves our moral consideration. Or if there is one, a vast majority of Americans, at least, seem unaware of it. Only a small percentage of Americans are vegetarians." (Would be funny if this would be written by an AI, as the dash seems to indicate).

Personally, I don't consider it my crusade to convince all these people that they're wrong and they should in fact be vegan and accept conscious AI morality. I feel more like a facilitator of the debate. That's one reason I'm not EA.

Thanks for engaging. I agree with quite a bit of what you're saying, although I do think that everyone's perspective is equally valid, fundamentally. In practical democracies there are many layers though between the raw public vote and a policy outcome. First, we mostly have representative democracy instead of direct democracy, then we have governments who have to engage with parliaments but also listen, to different extents, to scientists, opinion makers, and lobbyists. Everyone's perspective is valid, and in some questions (e.g. ethical ones) should imo be leading. However, in many practical policy decisions, it makes sense to also spend time listening to those who have thought longer about issues, and this mostly happens. Completely discarding people's perspectives is rude, bad, and likely leads to uprisings, I think.

I'd like consensus too but I'm afraid it leads to too indecisive governments. Works mostly in small groups I guess.

I agree with all your points of nuance.

I'm still having trouble parsing longtermists' thoughts about this issue. MacAskill does explicitly defend these two assumptions. He and others must understand where this leads?

I've spoken to many EA and rat longtermists, and while many were pragmatic (or simply never thought about this), some actually bit the bullet and admitted they effectively supported human extinction.

If people don't support human extinction, why do they not distance themselves from this outcome? I mean it would be easy: simply say, as imo a lower bar: yes we want to build many happy conscious AIs, but we do promise that if it's up to us, we'll leave earth alone.

I don't quite understand why longtermists are not saying this.

Would be nice if those disagreeing are saying why they're actually disagreeing

Nice post, guess I agree. I think it's even worse though: not only do at least some alignment researchers follow their own philosophy which is not universally accepted, it's also a particularly niche philosophy, and one that potentially leads to human extinction itself.

The philosophy in question is of course longtermism. Longtermism holds two controversial assumptions:

1. Symmetric population ethics: we have to create as much happy conscious life as possible. It's not just about making people happy, it's also about making happy people. In philosophy, and outside philosophy, most people think this is bonkers (I'm one of them).
2. Conscious AIs are morally relevant beings.

These two assumptions together lead to the conclusion that we must max out on creating conscious AIs, and that if these AIs end up in a resource conflict with humans (over e.g. energy, space, or matter), the AIs should be prioritized, since they can deliver most happiness per J, m^3 or kg. This leads to extinction of all humans.

I don't believe in ethical facts so even an ideology as, imo, bonkers as this one is not objectively false, I believe. However, I would really like alignment researchers and their house philosophers (looking at you, MacAskill) to distance themselves from extrapolating this idea all the way to human extinction. Beyond that bare minimum, I would like alignment researchers to start accepting democratic inputs in general.

Maybe democracy is the library you were looking for?

Sounds promising!

Somewhat related, there was an EA forum post recently about cost effectiveness of comms from OP. They calculated viewer minute per dollar, but I think conversions per dollar would be better. Would be interesting to compare the conversions per dollar you get with our data. Maybe good to post your approach there as a comment too?

Thanks for writing the post, automating xrisk-pilling people is really awesome, and more people should be trying to do it! Of course, traditional ways of automating x-pilling people are called 'books', 'media' and 'social media' and have been going strong for a while already. Still, if your chatbot works better, that would be awesome and imo should be supported and scaled!

We've done some research on xrisk comms using surveys. We defined conversion rate by asking readers the same open question before and after they consumed our intervention such as opeds or videos. The question we asked was: "List three events, in order of probability (from most to least probable), that you believe could potentially cause human extinction within the next 100 years." If people did not include AI or similar before, but did include it after our intervention, or if they raised AI's position in the top three, we counted them as converted. Conversion rates we got were typically in between 30% and 65% I think (probably decreasing over time). Paper here.

Maybe good to do the same survey for your chatbot? You can do so pretty easily with Prolific, we used n=300 and that's not horribly expensive. I'd be curious how high your conversion rates are.

Also, of course it's important how many people you can direct towards your website. Do you have a way to scale these numbers?

Keep up the good work!

I agree AI intelligence is and likely will remain spiky and some spikes are above human-level (of course a calculator also spikes above human-level). But I'm as of yet not convinced that the whole LLM-based intelligence spectrum will max out above takeover-level. But I'd be open for arguments.