Why are smart people probably dangerous? Because they can fool us. They are better than average people are at lying, and hiding their true intentions. Intentions that most likely relate to dominating the rest of us. (How could they not? I mean, wouldn't you use that brain-power to subjugate others...
Since people love doing this extrapolation/theorizing of "what if we gave what we thought was a simple command to an Artificial Super-Intelligence (ASI) and then it interpreted it wrong and tries to Kill All Humans as a result?!?": What if we programmed in — to be safe, at every level...
Ironically, I think it's because we can rationalize. And not "rationalize" in the sense of "coming up with excuses for acting on emotion" or whatnot. More in the sense of how we put stories to the maths, as it were. Reason. So ironically, logic is why we're so illogical. Perhaps...
We can't be sure that just the fear of being watched is enough to keep people on the straight and narrow. We have to actually watch them. All the time. Without some kind of embedded throttles or whatnot on AGI we can't be certain that it will not find an...
I'm assuming there are other people (I'm a person too, honest!) up in here asking this same question, but I haven't seen them so far, and I do see all these posts about AI "alignment" and I can't help but wonder: when did we discover an objective definition of "good"?...