MACannon

Wiki Contributions

Comments

Is the argument that AI is an xrisk valid?

Informally, it's the kind of intelligence (usually understood as something like " the capacity to achieve goals in a wide variety of environments") which is capable of doing that which is instrumental to achieving the goal. Given a goal, it is the capacity to achieve that goal, to do what is instrumental to achieving that goal. 

Bostrom, in Superintelligence (2014), speaks of it as  "means-end reasoning". 

So, strictly speaking, it does not involve reasoning about the ends or goals in service of which the intelligence/optimisation is being pressed.

Example: a chess-playing system will have some pre-defined goal and optimise instrumentally toward that end, but will not evaluate the goal itself.