You wake up tomorrow, and find out that super-smart and powerful aliens are arriving here in 10 years. We don't know anything about them other than the fact that they're super smart and powerful: e.g, their minds run on giant computers at THz clock speeds, they can integrate planets' worth of sensor data at once, they can do direct mass to energy conversion in their spaceships, they have weapons capable of destroying our planet in an instant, etc etc.

What do you, personally, do?

What should humanity overall do?

I think the answers to these questions shine a very important light on overlooked AGI alignment strategies. Rather than seeing the arrival of AGI as a technological change, we can see it as the arrival of a potentially competing / potentially collaborating species. Approximate intuition pump: humans :: AGI : animals :: humans -- if you were an emu (about to be hunted to extinction), how would you negotiate with the humans? If you were a monkey, how would you negotiate with the humans?

What are the frameworks we would use to negotiate with supersmart powerful aliens, or which animals might use to negotiate with people?

creating an ultimate ground rules moral framework: if we can create a moral framework which every species could agree to to improve the stable equilibrium of our interspecies game (where the lowest stable equilibrium is violence), then it might be convincing for an AGI coming out of our human values to maintain that ultimate ground rules moral framework. my best idea for that framework is some version of "information is sacred": I have no idea how exactly to measure information, but I think it all sort of ends up working out.
1. preservation of information: we must try to maintain as much information as possible. most of the universe could be compressed quite a bit, but not humans: our incompressability is what gives us moral worth. the same could be said of animals. how do you determine what can actually be compressed well? it's not physically possible to do perfectly, but you can have heuristics. this is also a good reason to keep animals in special wildlife conservation areas. unfortunately it might accord quite a high moral valence to hard drives: that's fine if humans still make it out in the end
2. continuation of computation: storing the mind state of a person isn't enough. you want their mind state to continue forwards. so somehow, we

...

Posts

Wikitag Contributions

Comments