This is a special post for quick takes by Q Home. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Imagine the classic paperclip maximizer thought experiment. We say AGI to make paperclips - AGI uses all matter of the Universe for it.
But now imagine a different version: we say AGI to "make paperclips that cost 1¢ (in human economy)". Now killing everyone isn't a solution: destroying humanity would destroy the economy.
Isn't it an interesting version of the thought experiment? Of course, everything can/will go wrong anyway, but maybe in a way funnier and more convoluted way. More funny and convoluted than "maximize human smiles", for example. Because AGI needs to take into account effects of a system (economic system), not just fulfill some fixed conditions.
I first mentioned the idea in this comment, a couple of people disagreed.
To take a step back: do you see a potential conceptual distinction between my idea and classic paperclip maximization? (Of course, you don't have to see it and/or agree that there's one. And even if there's one in theory it doesn't mean it exists in practice.)
Yes, it's always hard to define the "true reward" AI should strive for. But properties of the system "true reward + AI" may be easier to define.
[...]
If AI is able to reason/learn about properties of reward systems, then AI should be able to infer that taking 100% control over the reward system is a hack. Not something that can possibly be asked. So hacking the economy isn't just a solution "human doesn't expect" (some such solutions are very good), it's a solution that can't possibly be asked. This is one of the points of my idea: to introduce a distinction between unexpected solutions and nonsensical solutions.
6gwern
No. Not without a lot more work, because markets, evolution, gradient descent, Bayesian inference, and logical inference/prediction markets all have various isomorphisms and formal identities, which can make their 'differences' more a matter of nominalist preference, notation, and emphasis than necessarily any genuine conceptual distinction. You can define AIs which are quite explicitly architected as 'markets' of various sorts, like the 'Hayek machine' or the 'neural bucket brigade', or interpret them as natural selection if you prefer on agents with log utility (evolutionary finance), and so on; are those "markets", which can trade paperclips? Sure, why not.
I see that I need a post to at least explain myself. On the other hand, I worry to post too soon (maybe it's better to discuss something beforehand?). For the moment I decided to post this comment. I know, it's not formal, but I wanted to show what type of AI thinking I have in mind. And sorry for an annoying semantic nitpick ahead.
Not without a lot more work, because markets, evolution, gradient descent, Bayesian inference, and logical inference/prediction markets all have various isomorphisms and formal identities,