OpenAI was founded as a non-profit with a clear commitment to avoiding AI dangers and a unique structure to prevent investor control or excessive pressure from profit motives.
I want a strong mission lock to keep the company ethical and an AI ethics board with real teeth and power to enforce that.
But seeing past failures, it's obvious we have to be careful here. So I'm asking for ideas to make sure that "Prealign" or whatever the organisation ends up being called, actually does stay aligned.
I've been mainly thinking along conventional structural lines - maybe the ethics board should own the company, different share classes, incorporating in locations where a mission lock in the company's founding documents is respected and enforceable, choosing the right corporate design, etc.
Any specific idea of that sort is of value, as are unconventional or original ideas as well. I'll just share one unusual design thought I had: the pivot to profit.
When things look dangerous, pivot to immediate profit
The OpenAI situation highlighted that it's not just a question of the leadership holding out against immense financial and social pressures; you have to consider the employees and partners as well. If the company holds out but the employees jump ship[1] to a deep-pocketed partner or rival, carrying their knowledge with them, little will have achieved.
Now, in the normal course of operation, a company focused on deep research will turn up lots of designs, results, and customer needs that are valuable but not worth working on, because the deep research is the focus and priority.
So I suggest that if the deep research starts looking dangerous, the ethics board has the power to impound it and announce a pivot to immediate profit. The cutting edge research is taken off the table; and the company has to turn instead to monetising all those previous designs, results, and connections.
So instead of fighting profit head-on, it would go with it. Trading future profits and future immense risks for much more certain short-term cash. Now, people might certainly be disappointed (I know I would be) but it shouldn't provoke the all out fight that shutting down the company would.
Of course we'd want to only employ moral and dedicated people; but here's an omnipresent self-serving bias in everyone. It's easy to convince yourself (yes, you, yourself, and also me, myself), in ambiguous situations, that things are "not quite so bad" and that the AI ethics board are being unreasonable or overcautious. If the design depends on everyone being saintly and resistant to pressure, it will fail.
OpenAI was founded as a non-profit with a clear commitment to avoiding AI dangers and a unique structure to prevent investor control or excessive pressure from profit motives.
This failed.
I'm considering launching a commercial venture (possibly named "Prealign") to solve value generalisation and contribute to AI alignment, while avoiding anything that increases AI capabilities relative to alignment and values.
I want a strong mission lock to keep the company ethical and an AI ethics board with real teeth and power to enforce that.
But seeing past failures, it's obvious we have to be careful here. So I'm asking for ideas to make sure that "Prealign" or whatever the organisation ends up being called, actually does stay aligned.
I've been mainly thinking along conventional structural lines - maybe the ethics board should own the company, different share classes, incorporating in locations where a mission lock in the company's founding documents is respected and enforceable, choosing the right corporate design, etc.
Any specific idea of that sort is of value, as are unconventional or original ideas as well. I'll just share one unusual design thought I had: the pivot to profit.
When things look dangerous, pivot to immediate profit
The OpenAI situation highlighted that it's not just a question of the leadership holding out against immense financial and social pressures; you have to consider the employees and partners as well. If the company holds out but the employees jump ship[1] to a deep-pocketed partner or rival, carrying their knowledge with them, little will have achieved.
Now, in the normal course of operation, a company focused on deep research will turn up lots of designs, results, and customer needs that are valuable but not worth working on, because the deep research is the focus and priority.
So I suggest that if the deep research starts looking dangerous, the ethics board has the power to impound it and announce a pivot to immediate profit. The cutting edge research is taken off the table; and the company has to turn instead to monetising all those previous designs, results, and connections.
So instead of fighting profit head-on, it would go with it. Trading future profits and future immense risks for much more certain short-term cash. Now, people might certainly be disappointed (I know I would be) but it shouldn't provoke the all out fight that shutting down the company would.
How does that idea sound?
Of course we'd want to only employ moral and dedicated people; but here's an omnipresent self-serving bias in everyone. It's easy to convince yourself (yes, you, yourself, and also me, myself), in ambiguous situations, that things are "not quite so bad" and that the AI ethics board are being unreasonable or overcautious. If the design depends on everyone being saintly and resistant to pressure, it will fail.