Daniel Tan's Shortform

Daniel Tan

This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I increasingly view AI companies as the "ultimate model organism" of misalignment.

Frontier AI companies have been observed to exhibit many misalignment pathologies. Examples include: the talker not controlling the doer, avoiding external oversight and accountability, a tendency to seek power, and most of all a tendency to rationalise otherwise dubious behaviours as justified.

But I think the analogy runs much deeper. Frontier AI companies are in some sense the "ultimate scaffold" for the AIs they develop. As more and more work gets delegated to AI, the resources, clout, and even people within the organization are better thought of as tools and affordances for their most powerful models, rather than meaningfully in control of the processes. Insofar as we want to develop ecologically valid ways of understanding the alignment of language models, I think there is no better candidate for such analysis than the frontier AI companies.

Like AI models, frontier AI companies are (i) complex and hard to characterise, (ii) shaped by intense optimization pressures that inevitably affects their psychology, (iii) extremely opaque to scrutiny by third parties. Thus, alignment science (developed for models) may give us the tools we need to study the motivations and behaviours of frontier AI companies.

One such frame that transfers is the "alignment audit". This frame says that we should draft a list of principles defining behaviour we want, and then rigorously evaluate the entity in question on whether it holds this behaviour. If we were to apply the rigorous and exacting standards that we demand of AI to the very companies that create them, some might fail horrendously. An "alignment audit" of frontier AI companies' behaviour could very well conclude that they are not fit to be entrusted with making decisions that have a chance of ending in human extinction.

Another such frame is "AI control". This frame accepts that there is some probability that frontier AI companies will be scheming. As such, it's important to reduce the probability that potentially scheming entities can take irreversible catastrophic actions, by imposing strong transparency and monitoring requirements. We are currently nowhere near this state for frontier AI companies, and this is concerning to me.

Bottom line: I think many alignment researchers spend a lot of time thinking about alignment / monitoring for language models; I think it would be useful for them to think about the same for frontier AI companies.

In partial contrast to the above outlook (which I broadly agree with): starting from my general pessimistic view of the AGI situation, a vision that makes me feel a bit more optimistic is: thinking of each AI company as a being doing RSI, hopefully with a lizardbrain/[steering subsystem] made of its human employees piloting an increasingly capable exocortex made of hopefully "value-free" task AGIs ^[1] , hopefully only self-improving until it can get something pivotal done (such as making it so AGI tech is severely regulated globally).

like, these... (read more)

1dprkay1h

This feels like trying to use the AI safety hammer on the AI lab nail. I'm very skeptical of this since the AI safety memeplex has empowered labs to an astonishing degree. I'm very much reminded of Richard Ngo's alignment retrospective, and more tangentially The ML ontology and the alignment ontology. I was thinking: what in this post doesn't fit if you replace "frontier AI companies" with just "companies"? The difference seems explainable by people within the labs believing certain AI safety ideas, as well as other EA-adjacent ideas. I'm reminded of the only (other) large company founded by EAs: FTX. Going Infinite is a great story in itself, but what's utterly fascinating is that the author believes SBF, even after FTX collapses. This is the guy who wrote Liar's Poker and The Big Short, he should not be easy to bullshit! I think the question of what Effective Altruists were doing at a fraudulent cryptocurrency exchange of all places (or a company racing towards technology which might kill everyone) is unanswerable without an understanding that rationalists/EAs are extremely vulnerable to getting lied to in the face. After FTX went bankrupt, John Ray (who previously led the recovery of funds from Enron) took over as CEO and immediately cut through all the bullshit. A lot of this was just by using common sense, and understanding/values business/societal norms (like having a CFO). But a lot of it was being disagreeable and willing to view people as adversaries (while perhaps realizing the degree to which they're idiots/tools, and how the boundaries can be quite blurred or even nonexistent). My wish is for the alignment community to be a bit more like John Ray.

1MichaelLowe2h

" But I think the analogy runs much deeper. Frontier AI companies are in some sense the "ultimate scaffold" for the AIs they develop. As more and more work gets delegated to AI, the resources, clout, and even people within the organization are better thought of as tools and affordances for their most powerful models, rather than meaningfully in control of the processes. Insofar as we want to develop ecologically valid ways of understanding the alignment of language models, I think there is no better candidate for such analysis than the frontier AI companies." I am skeptical that this is a very helpful framing, beyond "large companies are hard to understand, just like large language models". First of all, one could argue that companies are just very aligned to the goal of making money (whereas it is pretty unclear what large languages are aligned to). Second, I don't see this post as presenting good arguments that companies are in fact just a tool for their models? I don't think it's true that Astra particularly cares about OpenAI IPOing successfully, or being released or not to the public, or more specifically that its outputs are shaped by these considerations. Finally and importantly, Altman and Amodei seem to have substantial control still, in a way that generic big company CEOs don't typically have. For example, my guess is if Amodei wanted, he could totally decide not to push frontier capabilities.