is a fine example of thinking you get when smart people do evil things and their minds come up with smart justifications why they are the heroes

I want to just agree because fuck those guys, but actually I think it's also shit justifications. A good justification might come from exploring the various cases where we have decided to not make something, analyzing them, and then finding that there's some set of reasons that those cases don't transfer as possibilities for AI stuff.

From the mechanise blogpost:

Nuclear weapons are orders of magnitude more powerful than conventional alternatives, which helps explain why many countries developed and continued to stockpile them despite international efforts to limit nuclear proliferation.

Yet the number of nuclear weapons in the world has decreased from its peak during the cold war. Furthermore, we've somehow stopped ourselves from using them, which suggests that some amount of steering is possible.

With regards to the blogpost as a whole, humanity fits their picture the most when it is uncoordinated and trapped in isolated clades, each of which is in a molochian red queen's race with the other, requiring people to rapidly upgrade to new tech if only to keep pace with their opponents in commerce or war. But this really isn't the only way we can organise ourselves. Many societies made do fairly well for long periods in isolation without "rising up the tech tree" (e.g. Japan post-sengoku jidai).

And even if it is inevitable... You can stop a car going at 60 mph by slowly hitting the brakes or by ramming it into a wall. Even if stopping is "inevitable", it does not follow that the wall and the gentle decceleration are identically preferable for the humans inside.

In practice, this likely boils down to a race. On one side are people trying to empower humanity by building coordination technology and human-empowering AI. On the other side are those working to create human-disempowering technology and render human labor worthless as fast as possible.
 

I mean if we're being completely candid here, there is almost no chance the first group wins this race right?

I think they centrally don't treat "automating all jobs in the economy" as ASI precursor, or threatening human extinction or permanent disempowerment.

From the post:

It has only been about one human generation since human cloning became technologically feasible. The fact that we have not developed it after only one generation tells us relatively little about humanity’s ability to resist technologies that provide immediate and large competitive advantages.

Human cloning enables essentially millions of Grothendiecks and von Neumanns, which is likely an immense advantage. Delaying ASI by one human generation (for a start) might actually be a very useful development. So this snippet probably doesn't intend to be giving an example analogous to ASI.

From later in the post:

The upside of automating all jobs in the economy will likely far exceed the costs, making it desirable to accelerate, rather than delay, the inevitable.

The point of delaying ASI is that it might allow humanity to change crucial details of the outcome of its development. Even with the premise of ASI being somehow inevitable, it leads to different consequences depending on how it's developed, which plausibly depends on when exactly it's developed, even if it's only a single human generation later than otherwise. So the relevant costs aren't costs of developing ASI, but relative costs of developing it earlier, when we know less about how to do that correctly, compared to developing it later.

But if "automating all jobs in the economy" is just a mundane technology that only threatens the current structure of society where most people have jobs (and so most of the costs are about the resulting societal upheaval), this snippet makes more sense. If the AI economy remains under humanity's control, there is much less path dependence to how introduction of this technology determines the outcome, and so it matters less for the eventual outcome if this happens soon vs. later.

A relevant point of information: Barnett (at least) is on record having some substantial motivating ethical commitments, which aren't mentioned in the post, regarding the moral value of AI takeover. (In short, my understanding is he thinks it's probably good because it's some agents getting lots of what they prefer, regardless of sentience status or specifics of those preferences. This description is probably missing heaps of nuance.)

I'd guess this is a very relevant question in the determination of how desirable it is to 'accelerate the [so-called] inevitable'^[1].

As far as I understand, at least one of the authors has an unusual moral philosophy such as not believing in consciousness or first-person experiences, while simultaneously believing that future AIs are automatically morally worthy simply by having goals.

[narrow point, as I agree with most of the comment]

For what it's worth, I think this seems to imply that illusionism (roughly, people who, in a meaningful sense, "don't believe in consciousness") makes people more inclined to act in ethically deranged ways, but, afaict, this mostly isn't the case, because I've known a few illusionists (was one myself until ~1 year ago) and, afaict, they were all decent people, not less decent than the average of my social surroundings.

To give an example, Dan Dennett was an illusionist and very much not a successionist. Similarly, I wouldn't expect any successionist aspirations from Keith Frankish.

There are caveats, though in that I do think that a sufficient combination of ideas which are individually fine, even plausibly true (illusionism, moral antirealism, ...), and some other stuff (character traits, paycheck, social milieu) can get people into pretty weird moral positions.

For an alternative view, look to FLF's fellowship among many other initiatives (including, I believe, some of Jan's) aiming to differentially accelerate human-empowering technology - especially coordination tech which might enable greater-than-before-seen levels of tech-tree steering!

I tried to just strong-downvote this and move on, but I couldn't. It's just too bad in too many ways, and from its scores it seems to be affecting too many people.

a fine example of thinking you get when smart people do evil things and their minds come up with smart justifications why they are the heroes

This is ad hominem in a nasty tone.

Upon closer examination it ignores key inconvenient considerations; normative part sounds like misleading PR.

Et tu quoque? Look at this next bit:

A major hole in the "complete technological determinism" argument is that it completely denies agency, or even the possibility that how agency operates at larger scales could change. Sure, humanity is not currently a very coordinated agent. But the trendline also points toward the ascent of an intentional stance. An intentional civilization would, of course, be able to navigate the tech tree. (For a completely opposite argument about the very high chance of a "choice transition," check https://strangecities.substack.com/p/the-choice-transition).

Maybe "agency at larger scales could change". I doubt it, and I think your "trendline" is entirely wishful thinking.

But even if it can change, and even if that trendline does exist, you're talking about an at best uncertain 100 or 500 year change. You seem to be relying on that to deal with a 10 to 50 year problem. The civilization we have now isn't capable of delaying insert-AI-consequence-here long enough for this "intentional" civilization to arise.

If the people you're complaining about are saying "Let's just build this and, what the heck, everything could turn out all right", then you are equally saying "Let's just hope some software gives us an Intentional Civilization, and what the heck, maybe we can delay this onrushing locomotive until we have one".

As for "complete technological determinism", that's a mighty scary label you have there, but you're still basically just name-calling.

On one side are people trying to empower humanity by building coordination technology and human-empowering AI.

Who? What "coordination technology"? How exactly is this "human-empowering AI" supposed to work?

As far as I can see, that's no more advanced, and even less likely to be feasible, than "friendly godlike ASI". And even if you had it, humans would still have to adapt to it, at human speeds.

This is supposed to give you an "intentional civilization" in time? I'm sorry, but that's not plausible at all. It's even less plausible than the idea that everything will just turn out All Right by itself.

... and that plan seems to be the only actual substance you're offering.

On the other side are those working to create human-disempowering technology and render human labor worthless as fast as possible.

This appears to assume that human labor should have value, which I assume to mean that it should be rewarded somehow, thus that performing such labor should accrue some advantage, other than having performed the labor itself... which seems to imply that people who do not perform such labor should be at a comparative disadvantage.

... meaning that other people have to work, on pain of punishment, to provide you and those who agree with you with some inchoately described sense of value.

If we're going to name-call ideas, that one sounds uncomfortably close to slavery.

It also seems to assume that not having to work is "disempowering", which is, um, strange, and that being "disempowered" (in whatever unspecified way) is bad, which isn't a given, and that most people aren't already "disempowered" right now, which would demand a very odd definition of what it means to be "disempowered".

... and the rest is just more ad hominem.

Second link is broken for me.