Normalization of deviance is an interesting and imo very underrated idea for studying how institutions composed of otherwise mostly reasonable people can still do things that from the outside seem quite obviously like own-goals. This is especially notable in the context of medium-low probability, high-consequence disasters. Importantly this seems to happen in many (most?) major institutions across government, business, healthcare, tech, and more^[1]. Counterintuitively, deviance and normalization of deviance is in fact the norm!

I particularly like Dan Luu's pull-quote here^[2]:

What these disasters typically reveal is that the factors accounting for them usually had “long incubation periods, typified by rule violations, discrepant events that accumulated unnoticed, and cultural beliefs about hazards that together prevented interventions that might have staved off harmful outcomes”. Furthermore, it is especially striking how multiple rule violations and lapses can coalesce so as to enable a disaster's occurrence.

Imo this is helpful contextualization to me for why AI companies take actions that seem neither guided by the long-term benefits of humanity nor clearly good for their bottom lines. I illustrated some of these dynamics with my story You're Absolutely Right, but imo I only scratched the surface.

How does normalization of deviance get "fixed"? In my limited historical reading, major safety-relevant cultural failures in industries and institutions typically only get patched after multiple major disasters. Tech is actually quite lucky in this regard: in other industries, many industry regulations and best practices were written in blood, but (eg) big tech companies like Google managed to become adequate at security after disasters that were only financially and optically costly, rather than directly costly in piles of corpses.

One hope I have in AI safety is that perhaps people both in and outside of the industry can beat the base rate and learn more from other people's mistakes, and/or update enough from just small disasters, instead of needing to wait on the major disasters. Another hope I have is that we'd see more major but not existential AI disasters (especially but not limited to loss-of-control incidents) before the AIs pose significant takeover threat, and that's enough to get the companies and/or external actors to take sufficiently drastic-but-correct actions for safety. Both are of course highly contingent, and far from guaranteed.

An unsentimental view of lab sociology is likely worth studying further, particularly by external people, with or without some degree of privileged access.

1. ^{^}

   I've also personally observed some of these dynamics in situations that are less like traditional "institutions", including student clubs and online communities like Wikipedia editors.

2. ^{^}

   I also like his general observations of tech companies on this dimension, stated in plain, unadorned but also not-PR'd language. He also recommends reading Banja, which I second.