Context: I like to approach various ML phenomena from the perspective of spatial transformations. Compared to the classic engineering approach, this approach provides such a sudden "bird's eye view" of what's happening in models that, after developing it, I felt like I had a double-descent by myself. Inspired by this...
As is well known, long-termism rests on three core assumptions: 1. The moral equality of generations. 2. The vast potential of the future. 3. The ability to influence the future. While the third assumption is commonly criticized, the first and second points receive far less attention for some reason, especially...
Context: I like to approach various ML phenomena from the perspective of spatial transformations. Compared to the classic engineering approach, this approach provides such a sudden "bird's eye view" of what's happening in models that, after developing it, I felt like I had a double-descent by myself. Inspired by this realization, I decided to write this post for those familiar with the basics but not deeply familiar with ML spaces in general.
Most key phenomena in ML can be understood as properties and dynamics of representations in spaces, rather than as algorithmic or statistical artifacts.
This perspective allows us to see common causes for seemingly unrelated effects. To get you started, I'll list some... (read 1569 more words →)
It seems to me that aligning a mesa-optimizer could be solved with the same methods used to align a base model. If so, distinction between "can create" and "can control" steps would be minimized. What concerns me more is how we would detect the existence of mesa-optimizers, especially if they emerge randomly.
Sorry for a long reply,
It looks like the outcome depends on the speed of technological enhancement and regulations. If the advantage of autofac grows faster than competitors can react (through violence, alliances, blockades), the system will collapse into a monopoly.
I personally believe that there is a greater than 0.5 chance that autofac will not be bound by any laws or regulations, because:
1. We currently cannot handle even simple problems in alignment.
2. Outside of alignment too.
As I have mentioned, a resource advantage is critical in such an economy, which is driven by widespread automation.
When agents unite, it eventually leads to the formation of an "equilibrium" that is actually unstable. Its imperfection will only be magnified by autofac, given that humans are not a significant resource. In other words, one group will ultimately become dominant in the long run. Since the group also had an initial imbalance, the larger agent will eventually gain a decisive advantage.
Of course, one can imagine a situation where groups can find a balance and constantly counterbalance each other, but from historical examples, we see that this is unlikely. Multipolar systems tend toward simplification, and pacts do not last long. It does not matter how many these groups are or how many agents are inside them: an imbalance will always exist, which automation will only amplify in the long term.
I generally agree that the autofac era seems like a logical next step. However, it seems it would dramatically accelerate capital consolidation, leading to an unprecedented gradual disempowerment.
In such an economy, a decisive resource advantage becomes equivalent to long-term global dominance. If this consolidation completes before ASI arrives, the first ASI will likely be built by an actor facing zero constraints on its deployment, which is a direct path to x-risk. This makes the prospect of a stable, "happy" pre-ASI autofac period seem highly doubtful.
(Though it's possible the dominant actor would choose to halt the race, that seems far from a given, even with total dominance.)
As is well known, long-termism rests on three core assumptions:
1. The moral equality of generations.
2. The vast potential of the future.
3. The ability to influence the future.
While the third assumption is commonly criticized, the first and second points receive far less attention for some reason, especially in the context of the most likely ASI development scenario. Talk of myriads of meaningful lives makes little sense if we stop imagining a utopian, densely populated galaxy and instead consider the motivations of the agent that will be shaping that galaxy.
In most development models, the first agent to achieve superintelligence (ASI) will become a singleton. Its behavior will, with high probability, be determined by instrumental convergence.
I don’t know if author will even see this comment, but beyond the mainstream theories of antagonistic pleiotropy and damage accumulation, the reader may find it interesting to explore the concept of programmed aging as an adaptation to combat pathogens. IMHO, this is a better example to defend the position than thymic involution, because it makes a much stronger case for the existence of a 'program'.