You could argue that a superintelligence would be efficient at all tasks as follows:
Assume that:
1. An AI will not knowingly be biased (if it knew it had a bias, it would correct it).
2. predicting the residual error of one's predictions is a task that superintelligences are definitionally better at than humans.
Then: superintelligences are efficient at all tasks.
The proof is by contradiction. Suppose a superintelligence has some residual error in some task that humans can predict. Then, by (2), the superintelligence can also predict that residual error. But (1) asserts that a superintelligence cannot know that it is biased, so a superintelligence cannot be efficient at any task.
I have seen very few arguments about superintelligence that rest on epistemic efficiency. Roughly speaking, epistemic efficiency with respect to X might be interpreted as "smarter in every way than X." But we usually talk about systems that are "smarter in some ways than humans." And the safety problem doesn't seem to change in a qualitative way at the threshold of "smarter in every way." Nor does economic value, or most indicators of interest. The only measure on which that's a fundamental threshold is "how hard it is for humans to do anything useful" (but this is not an indicator people are talking about if they talk about corporations, for obvious reasons...)
So while I might agree that a corporation is not a superintelligence on Nick's definition, this doesn't seem to have much bearing on the way in which the analogy is invoked in discussions. In general, this notion of superintelligence is a sufficient condition for lots of interesting phenomena, but not a necessary condition for almost anything.
It just seems like you semantically disagreeing about the use of the word "superhuman." This seems like a missed opportunity to help communicate about the value alignment problem. (Also, though it's not precisely related, I think that people really do think about AI better when they think about it as "idealized corporation" rather than "idealized human." Corporatization seems to be a better baseline than anthropomorphization, though neither is great.)
To make things more concrete, I think it is roughly as reasonable to ask about the "value alignment problem for organizations," and that many solutions to the value alignment problem for AI will also be applicable to organizations (and conversely, if someone came to me with an actually good proposal for value alignment for organizations, I would consider it worth-looking-at). Of course I think that value alignment problem for AI systems is much more important, and so where the two problems are disanalogous I care about the AI version (and also I want to reserve undecorated "value alignment" as a technical term referring to the AI version of the problem). But that judgment is unrelated to the epistemic efficiency of corporations---I'd think the same thing even if corporations were epistemically efficient, and I presume so would you.
Your basic complaint with people's use of corporations as an analogy really seems to be that AI systems will become very much more powerful, and that they will never have the same peculiar mix of abilities as human organizations.
(Indeed, assuming that an agent is smarter in every way simply makes the safety problem easier, and many of our disagreements about safety are based on me being willing to assume something like "epistemic efficiency with respect to an average college graduate," at least as a first step.)
It's an extremely plausible thing to expect given enough raw cognitive power.
It's a compact reply to people saying "An AI will do (believe) X because Y" when the AI is supposed to have a lot of cognitive power and a human could think of a better strategy (inference) for Y.
It has understandable precedents in market prices and in superhuman game-playing.
It's one of several compact properties that lets us distinguish important superintelligences from much weaker objects like corporations which people are tempted to shoehorn in as Objects of Equally Great Concern.
Efficiency is indeed a very powerful property and is often overpowered as an assumption for the conclusion being offered. On the other hand, it's also an important basic idea to understand because an AI which isn't efficient in any domain is not relevant (it must not even be able to play superhuman chess, if it's not efficient at chess).
Had a very visceral experience of feeling surrounded by a bunch of epistemically efficient (wrt me) agents in a markets game tonight. Just like "yup, I can choose to bet, or not bet, and if I do bet, I may even make money, because the market may well be wrong, but I will definitely, definitely lose money in expectation if I bet at all"
Boundedly rational ?means rational even when you don't have infinite computing power? Naturalistic ?refers to naturalized induction, where you're not a cartesian dualist who thinks your processes can't be messed with by stuff in the world and also you're not just thinking of yourself as a little black dot in the middle of Conway's game of life? Google says economic agent means one who has an impact on the economy by buying, selling or trading; I assign 65% to that being roughly the meaning in use here?
Somehow the epistemic efficiency thing reminds me of the halting problem; that whatever we try and do, it can just do it more. Or... somehow it actually reminds me more the other way, that it's solved the halting problem on us. Apologies for abuse of technical terms.
So an epistemically efficient agent, for example, is already overcoming all the pitfalls you see in movies of "not being able to understand the human drive for self sacrifice" or love, or etc.
Is there an analogue of efficient markets for instrumental efficiency? Some sort of master-strategy-outputting process that exists (or maybe plausibly exists in at least some special cases) in our world? Maybe Deep Blue at chess, I guess? Google maps for driving directions (for the most part)? reads to next paragraph. Well; not sure whether to update against Google Maps being an example from the fact that it's not mentioned in "instrumentally efficient agents are presently unknown" section
That said, "outside very limited domains" - well, I guess "the whole stock market, mostly" is a fair bit broader than "chess" or even "driving directions". Ah, I see; so though chess programs are overall better than humans, they're not hitting the "every silly-looking move is secretly brilliant" bar yet. Oh, and that's definitely not true of google maps - if it looks like it's making you do something stupid, you should have like 40% that it's in fact being stupid. Got it.
I can't tell if I should also be trying to think about whether there's a reasonable de
I seem to have found max comment length? Here's the rest:
I can't tell if I should also be trying to think about whether there's a reasonable definition of "the goals of google maps" wherein it actually is maximizing its goals right now in a way we can't advance. I don't think there is one?
I don't know why this hasn't happened to corporations - you'd think someone would try it, at some point, and that if it actually worked pretty well it would eventually allow them to outcompete, even if it was the sort of innovation that meant you had to climb uphill for a bit you'd expect people to keep periodically trying and for one of them eventually to overcome the activation energy barrier?
You could argue that a superintelligence would be efficient at all tasks as follows:
Assume that: 1. An AI will not knowingly be biased (if it knew it had a bias, it would correct it). 2. predicting the residual error of one's predictions is a task that superintelligences are definitionally better at than humans.
Then: superintelligences are efficient at all tasks.
The proof is by contradiction. Suppose a superintelligence has some residual error in some task that humans can predict. Then, by (2), the superintelligence can also predict that residual error. But (1) asserts that a superintelligence cannot know that it is biased, so a superintelligence cannot be efficient at any task.
Regarding corporations:
I have seen very few arguments about superintelligence that rest on epistemic efficiency. Roughly speaking, epistemic efficiency with respect to X might be interpreted as "smarter in every way than X." But we usually talk about systems that are "smarter in some ways than humans." And the safety problem doesn't seem to change in a qualitative way at the threshold of "smarter in every way." Nor does economic value, or most indicators of interest. The only measure on which that's a fundamental threshold is "how hard it is for humans to do anything useful" (but this is not an indicator people are talking about if they talk about corporations, for obvious reasons...)
So while I might agree that a corporation is not a superintelligence on Nick's definition, this doesn't seem to have much bearing on the way in which the analogy is invoked in discussions. In general, this notion of superintelligence is a sufficient condition for lots of interesting phenomena, but not a necessary condition for almost anything.
It just seems like you semantically disagreeing about the use of the word "superhuman." This seems like a missed opportunity to help communicate about the value alignment problem. (Also, though it's not precisely related, I think that people really do think about AI better when they think about it as "idealized corporation" rather than "idealized human." Corporatization seems to be a better baseline than anthropomorphization, though neither is great.)
To make things more concrete, I think it is roughly as reasonable to ask about the "value alignment problem for organizations," and that many solutions to the value alignment problem for AI will also be applicable to organizations (and conversely, if someone came to me with an actually good proposal for value alignment for organizations, I would consider it worth-looking-at). Of course I think that value alignment problem for AI systems is much more important, and so where the two problems are disanalogous I care about the AI version (and also I want to reserve undecorated "value alignment" as a technical term referring to the AI version of the problem). But that judgment is unrelated to the epistemic efficiency of corporations---I'd think the same thing even if corporations were epistemically efficient, and I presume so would you.
Your basic complaint with people's use of corporations as an analogy really seems to be that AI systems will become very much more powerful, and that they will never have the same peculiar mix of abilities as human organizations.
(Indeed, assuming that an agent is smarter in every way simply makes the safety problem easier, and many of our disagreements about safety are based on me being willing to assume something like "epistemic efficiency with respect to an average college graduate," at least as a first step.)
The point of 'efficiency' is that:
Efficiency is indeed a very powerful property and is often overpowered as an assumption for the conclusion being offered. On the other hand, it's also an important basic idea to understand because an AI which isn't efficient in any domain is not relevant (it must not even be able to play superhuman chess, if it's not efficient at chess).
Had a very visceral experience of feeling surrounded by a bunch of epistemically efficient (wrt me) agents in a markets game tonight. Just like "yup, I can choose to bet, or not bet, and if I do bet, I may even make money, because the market may well be wrong, but I will definitely, definitely lose money in expectation if I bet at all"
Boundedly rational ?means rational even when you don't have infinite computing power? Naturalistic ?refers to naturalized induction, where you're not a cartesian dualist who thinks your processes can't be messed with by stuff in the world and also you're not just thinking of yourself as a little black dot in the middle of Conway's game of life? Google says economic agent means one who has an impact on the economy by buying, selling or trading; I assign 65% to that being roughly the meaning in use here?
Somehow the epistemic efficiency thing reminds me of the halting problem; that whatever we try and do, it can just do it more. Or... somehow it actually reminds me more the other way, that it's solved the halting problem on us. Apologies for abuse of technical terms.
So an epistemically efficient agent, for example, is already overcoming all the pitfalls you see in movies of "not being able to understand the human drive for self sacrifice" or love, or etc.
Is there an analogue of efficient markets for instrumental efficiency? Some sort of master-strategy-outputting process that exists (or maybe plausibly exists in at least some special cases) in our world? Maybe Deep Blue at chess, I guess? Google maps for driving directions (for the most part)? reads to next paragraph. Well; not sure whether to update against Google Maps being an example from the fact that it's not mentioned in "instrumentally efficient agents are presently unknown" section
That said, "outside very limited domains" - well, I guess "the whole stock market, mostly" is a fair bit broader than "chess" or even "driving directions". Ah, I see; so though chess programs are overall better than humans, they're not hitting the "every silly-looking move is secretly brilliant" bar yet. Oh, and that's definitely not true of google maps - if it looks like it's making you do something stupid, you should have like 40% that it's in fact being stupid. Got it.
I can't tell if I should also be trying to think about whether there's a reasonable de
I seem to have found max comment length? Here's the rest:
I can't tell if I should also be trying to think about whether there's a reasonable definition of "the goals of google maps" wherein it actually is maximizing its goals right now in a way we can't advance. I don't think there is one?
I don't know why this hasn't happened to corporations - you'd think someone would try it, at some point, and that if it actually worked pretty well it would eventually allow them to outcompete, even if it was the sort of innovation that meant you had to climb uphill for a bit you'd expect people to keep periodically trying and for one of them eventually to overcome the activation energy barrier?