Jan_Kulveit's Shortform

Jan_Kulveit

This is a special post for quick takes by Jan_Kulveit. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Crossposting: [recent Mechanize blogpost advocating for extreme technological determinism and speeding up human disempowerment] is a fine example of thinking you get when smart people do evil things and their minds come up with smart justifications why they are the heroes. Upon closer examination it ignores key inconvenient considerations; normative part sounds like misleading PR.

A major hole in the "complete technological determinism" argument is that it completely denies agency, or even the possibility that how agency operates at larger scales could change. Sure, humanity is not currently a very coordinated agent. But the trendline also points toward the ascent of an intentional stance. An intentional civilization would, of course, be able to navigate the tech tree. (For a completely opposite argument about the very high chance of a "choice transition," check https://strangecities.substack.com/p/the-choice-transition).

In practice, this likely boils down to a race. On one side are people trying to empower humanity by building coordination technology and human-empowering AI. On the other side are those working to create human-disempowering technology and render human labor worthless as fast as possible.

My guess is when people stake their careers and fortune and status on the second option, their minds will work really hard to not see the choice.

Also: at least to me, the normative part sounds heavily PR sanitized, with obligatory promises of "medical cures" but shiying away from explaining either what would be the role of humans in the fully automated economy, or the actual moral stance of the authors. As far as I understand, at least one of the authors has an unusual moral philosophy such as not believing in consciousness or first-person experiences, while simultaneously believing that future AIs are automatically morally worthy simply by having goals. This philosophy leads them to view succession by arbitrary AI agents as good, and the demise of humans as not a big deal.

...benign scenarios in which AIs get legal rights and get hired to run our society fair and square. A peaceful AI takeover would be good, IMO.
...humans willingly transfer power to AIs through legal and economic processes. I think this second type will likely be morally good, or at least morally neutral.

Why do you believe this? For my part, one of the major ruinous scenarios on my mind is one where humans delegate control to AIs that then goal-misgeneralize, breaking complex systems in the process; another is one where AIs outcompete ~all human economic eff... (read more)

11Matthew Barnett1y

I'd consider it bad if AIs take actions that result in a large fraction of humans becoming completely destitute and dying as a result. But I think such an outcome would be bad whether it's caused by a human or an AI. The more important question, I think, is whether such an outcome is likely to occur if we grant AIs legal rights. The answer to this, I think, is no. I anticipate that AGI-driven automation will create so much economic abundance in the future that it will likely be very easy to provide for the material needs of all biological humans. Generally I think biological humans will receive income through charitable donations, government welfare programs, in-kind support from family members, interest, dividends, by selling their assets, or by working human-specific service jobs where consumers intrinsically prefer hiring human labor (e.g., maybe childcare). Given vast prosperity, these income sources seem sufficient to provide most humans with an adequate, if not incredibly high, standard of living.

4loic1y

Thanks for the reply, it was helpful. I elaborated my perspective and pointed out some concrete disagreements with how labor automation would play out, I wonder if you can identify the cruxes in my model of how the economy and automated labor interact. I'd frame my perspective as; "We should not aim to put society in a position where >90%+ of humans need government welfare programs or charity to survive while vast numbers of automated agents perform the labor that humans are currently depending on to survive." I don't believe we have the political wisdom or resilience to steer our world in this direction while preserving good outcomes for existing humans. We live in a something like a unique balance where through companies, the economy provides individuals the opportunity to sustain themselves and specialize while contributing to a larger whole which typically provides goods and services which benefit other humans. If we create digital minds and robots to naively accelerate these emergent corporate entities' abilities to generate profit, we lose an important ingredient in this balance, human bargaining power. Further, if we had the ability to create and steer powerful digital minds (which is also contentious), it doesn't seem obvious that labor automation is a framing that would lead to positive experiences for humans or the minds. [...] I'm skeptical that economic abundance driven by automated agents will by default manifest as an increased quality and quantity of goods and services enjoyed by humans, and that humans will continue to have the economic leverage to incentivize these human specific goods [...] I expect the amount of roles/tasks available where consumers prefer hiring humans is a rounding error compared to the amount of humans that depend on work

Jan_Kulveit's Shortform

7

Ω 5

Jan_Kulveit's Shortform

7

Ω 5