Epistemics: I've been ticking over the idea of a politically-naive Technical Safety field for a while, and while rough, this post enapsulates my main concerns with the field's direction.
Here's Richard Ngo's original post, to which I am responding. I posted a version of this post in commentary on there. Thank you Richard for sharing your provocative thinking.
Ngo's proposed solution
I took away from Richard's post that he believes previous community norms have failed to support the community's goals, and that is due to the influence of power and money. He notes AI is undergoing civilisation-scale investment and argue we can address the field's failures by updating community norms.
I don't think this is the right path
An alternative solution is to work with the money and power. The political and capital arms of the field can draw on an intellectual arm, where we can have nice norms. In the political and capital arms it might be better to have some realism - where the rubber hits the road you want to have some grip.
It really seems imperative to avoid a turning-inwards as safety has now its biggest opportunity of all - the political will to act is what will restrict for-profits and institute the controls we need to ride this out, or arrest it. An intellectual renaissance means little if nobody's paying attention, or if it provides tools that nobody wants to use, or if - as seems most likely now - new catastrophes pop up and safety doesn't have the tools to deal with them.
That will be a true disaster for us: if there is an opportunity to slow down the race, and resourcing has been turned away from the levers which would allow legislators to enforce the obviously-needed pause.
Safety should be split, but the intellectual arm shouldn't shut its eyes
Perhaps we can acknowledge the split between the three arms and find mechanisms to give the intellectual arm authority over the other two - doing the work of selling the fruit of the intellectual to the political and capital arms. Selectively meeting the demands of those other arms of the field, acknowledging that they are not interested in the same thing, may give us the clarity of mind and purity of mission without another intellectual turning-inwards.
Avoiding mass death and economic catastrophe is a pretty easy sell if you think it's coming soon. Capital markets and insurance price hazards. Governments have a mandate to avoid instability. If we look to solve the problem by addressing real power then we need to talk to people from the nuclear and biotechnology industries who have experience with catastrophic risk and regulation.
Making it concrete, in the face of regulatory capture
We should talk to people from financial systemic risk and epidemiology to understand how to make social-scale risk arguments.
We need people working out how to internalise the externalities through regulation and enforcement, and other people working to sell these measures to policymakers and executives. A lot of this can be informed by history.
Some of these instruments have been captured, but a flawed regulatory body or economic mechanism still exerts more influence on the real world than a better-normed blog post shared around what is essentially an academic community.
Nonetheless, capture points out the need for methods compatible with corporate and political incentives, made concrete, that can be verifiably enforced, rather than idealistic gesturing about making things safe or more, "trust-the-company," Ngo rightly critiques.
In summary
Safety's insistence on treating the problem as an intellectual and scientific one, without regard to the messy realities, has been to its detriment so far. Politics happens to everyone, whether they're in it or not. I also agree that we can point the finger for these failures squarely at political naivety and the community's refusal to interface with real-world power dynamics. There was no chance, outside fast takeoff, that we'd get anywhere near superintelligent AI without a massive influx of capital and political wrestling.
Ngo correctly diagnoses this mistake and then repeat it in his solutions. Norms have a place alongside that as part of the intellectual community but they will have little effect on their own. They must be matched with a realistic approach to power, capital, and politics.
In contrast, by taking a realpolitik approach, technical safety can carve out its own intellectual tradition while still providing real-world value. It is time for the community to get real, and resource appropriately, avoiding the turning-inwards to idealism that Ngo proposes will repair alignment.
Epistemics: I've been ticking over the idea of a politically-naive Technical Safety field for a while, and while rough, this post enapsulates my main concerns with the field's direction.
Here's Richard Ngo's original post, to which I am responding. I posted a version of this post in commentary on there. Thank you Richard for sharing your provocative thinking.
Ngo's proposed solution
I took away from Richard's post that he believes previous community norms have failed to support the community's goals, and that is due to the influence of power and money. He notes AI is undergoing civilisation-scale investment and argue we can address the field's failures by updating community norms.
I don't think this is the right path
An alternative solution is to work with the money and power. The political and capital arms of the field can draw on an intellectual arm, where we can have nice norms. In the political and capital arms it might be better to have some realism - where the rubber hits the road you want to have some grip.
It really seems imperative to avoid a turning-inwards as safety has now its biggest opportunity of all - the political will to act is what will restrict for-profits and institute the controls we need to ride this out, or arrest it. An intellectual renaissance means little if nobody's paying attention, or if it provides tools that nobody wants to use, or if - as seems most likely now - new catastrophes pop up and safety doesn't have the tools to deal with them.
That will be a true disaster for us: if there is an opportunity to slow down the race, and resourcing has been turned away from the levers which would allow legislators to enforce the obviously-needed pause.
Safety should be split, but the intellectual arm shouldn't shut its eyes
Perhaps we can acknowledge the split between the three arms and find mechanisms to give the intellectual arm authority over the other two - doing the work of selling the fruit of the intellectual to the political and capital arms. Selectively meeting the demands of those other arms of the field, acknowledging that they are not interested in the same thing, may give us the clarity of mind and purity of mission without another intellectual turning-inwards.
Avoiding mass death and economic catastrophe is a pretty easy sell if you think it's coming soon. Capital markets and insurance price hazards. Governments have a mandate to avoid instability. If we look to solve the problem by addressing real power then we need to talk to people from the nuclear and biotechnology industries who have experience with catastrophic risk and regulation.
Making it concrete, in the face of regulatory capture
Some of these instruments have been captured, but a flawed regulatory body or economic mechanism still exerts more influence on the real world than a better-normed blog post shared around what is essentially an academic community.
Nonetheless, capture points out the need for methods compatible with corporate and political incentives, made concrete, that can be verifiably enforced, rather than idealistic gesturing about making things safe or more, "trust-the-company," Ngo rightly critiques.
In summary
Safety's insistence on treating the problem as an intellectual and scientific one, without regard to the messy realities, has been to its detriment so far. Politics happens to everyone, whether they're in it or not. I also agree that we can point the finger for these failures squarely at political naivety and the community's refusal to interface with real-world power dynamics. There was no chance, outside fast takeoff, that we'd get anywhere near superintelligent AI without a massive influx of capital and political wrestling.
Ngo correctly diagnoses this mistake and then repeat it in his solutions. Norms have a place alongside that as part of the intellectual community but they will have little effect on their own. They must be matched with a realistic approach to power, capital, and politics.
In contrast, by taking a realpolitik approach, technical safety can carve out its own intellectual tradition while still providing real-world value. It is time for the community to get real, and resource appropriately, avoiding the turning-inwards to idealism that Ngo proposes will repair alignment.