This post was rejected for the following reason(s):
Low Quality or 101-Level AI Content. With the recent large influx of new users interested in AI, we are raising the bar for submission from new users about AI. To approve content, we generally require it to have some mix of reasoning well, making a new and interesting point, bringing new evidence, and/or building upon prior discussion. If your submission was rejected for this reason, possibly you want to spend more reading existing material. The AI Intro Material wiki-tag is a good place, for example.
It seems to me that one of the problems facing the whole space is a lack of any bounds for misalignment scenarios.
It could be helpful to have clear impossible and possible baskets to build a clearer set of objectives to work through.
An example might be. Can a pure gray goo physically escape our atmosphere, or would it need to use a launch system.
It gives a solvable bound on how the system would embody itself if it values survival in the long term.... Which any runaway computronium scenario would seem to imply. If space by launch system is a bound, then the runaway is restricted until after use of launch system... Which gives us new instrumentals
If we can't say anything about behavioral bounds based on physical bounds then... The problem becomes something like asking people to work on making ghost or leprechaun traps....
Discovering and controlling any, or ideally many, of these essentially... "great AGI filters" could be a way of adding layers of... instrumental control if that's a term