https://en.wikipedia.org/wiki/Lump_of_labour_fallacy we have many cases where adding labor-saving automation or increasing the size of the workforce did not increase unemployment at all. there are also a few counterexamples.
"robust" question: the result does not change if a small fraction of oracle queries are modified. if the oracle does a spot check of claims, that's good enough. (where the "oracle" represents human judgment, this had better be true, because human judgments are not perfectly reliable!)
the verifier asks the unreliable prover to make queries to an oracle, and the verifier checks that those answers are correct. the verifier can make direct oracle calls itself, but only very few.
Agents as "proxy consumers", making purchases on behalf of humans; what kinds of price distortions could imperfect approximations of human preferences cause?
competition might intensify since bots are better at comparison shopping
Agents as "matchmakers" in two-sided markets, again as imperfect proxies for humans' preferences
in high dimensions of match quality, this can lead to worse matches even if errors are small
AIs can collude on price-setting
AI agents empirically collude a lot
Game theory of AI agents may be different than game theory that imagines the units to be Homo economicus humans
AI agents that can condition actions on each other's source code, or manipulate their own memories, have coordination options not available to humans
AI agents may manipulate their own future preferences, or resist external attempts to manipulate their preferences
Firms may be different in the world of AI agents.
will firm sizes get bigger because coordination is easier?
will AI agents collude with each other at the expense of the firm, more effectively than human workers?
how could agents "reward" or "incentivize" each other? humans usually use money; we'd need new record-keeping or units of exchange to do the same for bots.
what kinds of systemic fragility (financial crashes, supply chain failures) should we worry about due to bots making correlated errors?
what kinds of institutions do we want for agents
identity, registration, record-keeping
legal fictions like corporate personhood for agents, so they could have assets in their own "name"?
should we penalize or blacklist agents that engage in bad behavior like contract violations?
RLHF implicitly optimizes the sum of rewards over time; if the person's values change over time as a result of interacting with the AI, that means the AIs are incentivized to change the person into the kind of person who will give out a lot of rewards, which can be undesirable (eg you could turn them into an addict, who gives upvotes very reliably when they get their fix.) on the other hand, locking in a present utility function is not necessarily better.
straightforward norms will be exploited; if false product reviews are penalized, agents will find a way to penalize their competitors.
add an (AI) verifier to check the credibility of accusations will cut down on abuse of norms, but is also computationally expensive. reputation-weighted reporting of norm violations is cheaper and works better. the system penalizes false reports, and in particular penalizes them more than it rewards true reports, by an increasing factor as the # of reports increases.
if agents are playing an unknown game, you can learn their utility functions based on giving payments and signals to the players and seeing how this changes their play.
https://www.nature.com/articles/s41586-026-10549-w "Evaluating large language models for accuracy incentivizes hallucinations." It does. accuracy scores mean it's always better to guess than to say you don't know.