LESSWRONG
LW

1099
kapedalex
2350
Message
Dialogue
Subscribe

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
No wikitag contributions to display.
3How singleton contradicts longtermism
2mo
1
1About corrigbility and thrustfulness
2mo
0
1Probably Overrated Approaches in AI Alignment
2mo
0
Alignment Problems All the Way Down
kapedalex8d10

It seems to me that aligning a mesa-optimizer could be solved with the same methods used to align a base model. If so, distinction between "can create" and "can control" steps would be minimized. What concerns me more is how we would detect the existence of mesa-optimizers, especially if they emerge randomly.

Reply
The Autofac Era
kapedalex12d10

Sorry for a long reply,
It looks like the outcome depends on the speed of technological enhancement and regulations. If the advantage of autofac grows faster than competitors can react (through violence, alliances, blockades), the system will collapse into a monopoly.
I personally believe that there is a greater than 0.5 chance that autofac will not be bound by any laws or regulations, because:

1. We currently cannot handle even simple problems in alignment.
2. Outside of alignment too.

Reply
The Autofac Era
kapedalex2mo10

As I have mentioned, a resource advantage is critical in such an economy, which is driven by widespread automation.

When agents unite, it eventually leads to the formation of an "equilibrium" that is actually unstable. Its imperfection will only be magnified by autofac, given that humans are not a significant resource. In other words, one group will ultimately become dominant in the long run. Since the group also had an initial imbalance, the larger agent will eventually gain a decisive advantage.

Of course, one can imagine a situation where groups can find a balance and constantly counterbalance each other, but from historical examples, we see that this is unlikely. Multipolar systems tend toward simplification, and pacts do not last long. It does not matter how many these groups are or how many agents are inside them: an imbalance will always exist, which automation will only amplify in the long term.

Reply
The Autofac Era
kapedalex2mo10

I generally agree that the autofac era seems like a logical next step. However, it seems it would dramatically accelerate capital consolidation, leading to an unprecedented gradual disempowerment.

In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, "happy" pre-ASI autofac period seem highly doubtful.

(Though it's possible the dominant actor would choose to halt the race, that seems far from a given, even with total dominance.)

Reply
Longevity and the Mind
kapedalex2mo10

I don’t know if author will even see this comment, but beyond the mainstream theories of antagonistic pleiotropy and damage accumulation, the reader may find it interesting to explore the concept of programmed aging as an adaptation to combat pathogens. IMHO, this is a better example to defend the position than thymic involution, because it makes a much stronger case for the existence of a 'program'.

Reply