is it really reasonable to have a full explicit list of problems that might arise, and how likely you think they are, and an estimated optimal amount of investment in prevention/preparedness? this seems like one of those fundamental problems with "doing Bayesianism" in practice, where "things that could go wrong" is not an enumerable list, and where estimates are often done by feel.
"don't worry about problems till they occur, and then react bigly when they do" is obviously bad in some contexts (like, there should be departments of the government and some companies that plan for natural disasters, and certain predictable risks like disease are worth taking some actions to prevent as an individual) but i'm pretty sympathetic to doing this for things that have literally never happened before. the human mind is just really, really flawed. calling a thing "science fiction" and blowing it off until it actually happens, and then taking it Super Seriously and taking over the response, is just how practical people behave, and they may have a point.
the plans you make before things are Really Happening are often...bad. you are not in the same frame of mind you would be if it was Really Happening.
it is really hard to tell whose predictions about the future are realistic, until they're actually happening.
if I look at who's been rightest about AI, it definitely wasn't who I thought. it wasn't the people who had the most experience, or who seemed to make the most logical sense, or who were approaching things in the most rigorous fashion, or who seemed to be most ethically/philosophically trustworthy. it *was*, interestingly, the people who had the most raw brainpower.
if I look at who was rightest about COVID, it also wasn't who I thought. it was Robin Hanson, who predicted that COVID would become endemic because containing it was more costly than the disease itself. that sounded crazypants at the time.
it is psychologically more manageable to only worry about stuff that you actually need to act on. a bunch of hypothetically-we-may-need-to stuff hanging in the air is stressful and promotes unproductive dithering.
some people can actually operate well in planning/future-prediction/hypothetical mode, but not everybody can, including many people whose skills will be useful for the response. Good "doers" are often uncomfortable with ambiguity, or have a practice of deliberately tuning it out and focusing only on the concrete and "real".
https://arxiv.org/abs/2407.18074 "Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts", Dima Ivanov, Inbal Talgam-Cohen algorithm that iteratively optimizes principal & agent policies where the principal offers rewards to the agent based on outcomes.
https://www.cs.cmu.edu/~conitzer/decisionWINE20.pdf Decision Scoring Rules Caspar Oesterheld, Vincent Conitzer, how should a principal pay an agent for good recommendations? shares in the success of the project.
https://ojs.aaai.org/index.php/AAAI-SS/article/view/42904/50464 some empirical experiments where prover-estimator debate games do not converge towards getting the right answer. despite RL training on accuracy, the estimator does not get more accurate over time; despite RL training on convincingness, the prover's reward does not improve over time. one cannot assume that a natural reward structure will actually incentivize more accurate estimators or more persuasive provers. natural language argument might just be a hard domain to RL in.
"The central feature of agency for our purposes is that agents are systems whose outputs are moved by reasons (Dennett, 1987). In other words, the reason that an agent chooses a particular action is that it “expects it” to precipitate a certain outcome which the agent finds desirable...Systems whose actions are moved by reasons, are systems that would act differently if they “knew” that the world worked differently."
this is very close to my view!
it necessitates causality and counterfactuals so they do Pearl-inspired models.
it excludes RL agents, which "would only pursue a different policy if retrained in a different environment." however the RL training process may be an agent!