Forecasting Infrastructure

Wiki Contributions



First off thank you for writing this, great explanation.

  • Do you anticipate acceleration risks from developing the formal models through an open, multilateral process? Presumably others could use the models to train and advance the capabilities of their own RL agents. Or is the expectation that regulation would accompany this such that only the consortium could use the world model?
  • Would the simulations be exclusively for 'hard science' domains - ex. chemistry, biology - or would simulations of human behavior,  economics, and politics also be needed? My expectation is that it would need the latter, but I imagine simulating hundreds of millions of intelligent agents would dramatically (prohibitively?) increase the complexity and computational costs.

This seems like an important crux to me, because I don't think greatly slowing AI in the US would require new federal laws. I think many of the actions I listed could be taken by government agencies who over-interpret their existing mandates given the right political and social climate. For instance, the eviction moratorium during COVID, obviously should have required congressional action, but was done by fiat through an over-interpretation of authority by an executive branch agency. 

What they do or do not do seems mostly dictated by that socio-political climate, and by the courts, which means less veto points for industry.


I agree that competition with China is a plausible reason regulation won't happen; that will certainly be one of the arguments advanced by industry and NatSec as to why it should not be throttled. However, I'm not sure, and currently don't think it will, be stronger than the protectionist impulses,. Possibly it will exacerbate the "centralization" of AI dynamic that I listed in the 'licensing' bullet point, where large existing players receive money and de-facto license to operate in certain areas and then avoid others (as memeticimagery points out). So for instance we see more military style research, and GooAmBookSoft tacitly agree to not deploy AI that would replace lawyers.


To your point on big tech's political influence; they have, in some absolute sense, a lot of political power, but relatively they are much weaker in political influence than peer industries. I think they've benefitted a lot from the R-D stalemate in DC; I'm positing that this will go around/through this stalemate, and I don't think they currently have the softpower to stop that.


hah yes - seeing that great post from johnwentsworth inspired me to review my own thinking on RadVac. Ultimately I placed a lower estimate on RadVac being effective - or at least effective enough to get me to change my quarantine behavior - such that the price wasn't worth it, but I think I get a rationality demerit for not investing more in the collaborative model building (and collaborative purchasing) part of the process.


I'm sorry I didn't see this response until now - thank you for the detailed answer!


I'm guessing your concern feels similar to ones you've articulated in the past around... "heart"/"grounded" rationality, or a concern about "disabling pieces of the epistemic immune system". 

I'm curious if 8 mo's later you feel you can better speak to what you see as the crucial misunderstanding?


Out of curiosity what's one of your more substantive disagreements with Thiel?


Forecast - 25 mins

  • I thought it was more likely that in the short run there could be a preference cascade among top AGI researchers, and as others have mentioned due to the operationalization of top AGI researchers might be true already.
  • If this doesn't become a majority concern by 2050, I expect it will be because of another AI Winter, and I tried to have my distribution reflect that (a little hamfistedly).

Thanks for posting this. I recently reread the Fountainhead, which I similarly enjoyed and got more out of than did my teenage self - it was like a narrative, emotional portrayal of the ideals in Marc Andreessen's It's Time to Build essay.

I interpreted your section on The Conflict as the choice between voice and exit.

Load More