At the risk of embarrassing myself, I’ll share a confession.

For context, I took five years of Latin: four in high school and one in college. In addition to learning the language, all my Latin classes taught a lot about Roman history. Emperors, internal politics, Caesar, etc. I was always learning some random bag of facts about Roman history. In high school, I won the award for top Latin student in my graduating class. So I wasn’t a bad Latin student.

Here’s the confession: I somehow don’t even vaguely remember the rough timespan the Roman Empire existed. Maybe Jesus time? I know he was killed by the Romans (is that right?). Were they around for a long time after? A long time before that? When was Romulus and Remus allegedly fighting? Virgil wrote the Aeneid when? I don’t have a clue. Despite being a kind of “Latin expert” I am missing a much more important foundational fact: when all of this was happening.

When I say trees are made out of air I’m not talking about the fact that there is a lot of empty space inside a tree (or actually anything made out of atoms). I mean something more mind-blowing. 

Imagine you are holding a piece of dry wood. Where did that wood come from? A tree. Okay sure, but what are you actually holding? Where did the tree get that stuff? It turns out that almost all of the mass in dry wood comes from the  in the air. This follows from a simple fact about how photosynthesis works:

The carbon and the oxygen in the glucose () come from the  and the hydrogen comes from the water. Since hydrogen is the lightest atom in the universe, it adds almost no mass, so nearly all the mass of glucose traces back to the . Wood is mostly built from glucose, so by mass, a dead tree is mostly the carbon and oxygen that came out of the air.

It’s kind of unintuitive. Wood is hard and stiff and air is not even close to either of those things. But if you are a biologist or know some basic chemistry you would think that this is the kind of obvious thing you would want to know. At the very least you would expect the best math and science students in the world to be familiar with such basic biology. But take a look at this video of a documentary film crew asking MIT graduates where they think wood comes from:

One student, upon being told that most of the mass in the wood comes from  and not the dirt in the ground, said “that's very disturbing and I wonder how that could happen.” And that’s an MIT graduate.

“Trees come from mostly air” is pretty fundamental for biology because it follows from photosynthesis which is in many ways the basis for life on earth. In a sane world, every 8th grader would know that dry wood is mostly , not like minerals found in the ground or something. MIT grads probably have vast amounts of detailed knowledge of math, physics, biology, chemistry or all of the above. But they don’t know some basic facts about biology. 

There is this assumption that you should first learn the foundational facts of an area of study and then move to more and more specific questions and ideas. As you move up levels of classes, the foundational stuff seems more and more basic and less and less relevant. Some stuff is continually hammered in because it’s useful background knowledge. In a perfect world, this helps you internalize the basics and learn how to reason about them to solve harder and harder problems.

There are some areas where, without much effort, this may work out. For example, you learn fractions in first or second grade but will probably understand them more deeply by the time you get to calculus because you need to know fractions to do calculus and all the classes that come before calculus. But not all foundational knowledge is like this! Knowing where trees come from doesn’t help you answer organic chemistry or evolutionary biology questions. 

Being 1. foundational and being 2. useful for answering increasingly specific questions are different things. They are certainly not orthogonal but they are also not perfectly correlated. When 1 & 2 diverge, you get MIT grads who are confused about what wood is.

In AI safety, this can be a serious problem.

I have had one-on-ones or interviewed dozens of students who want a career in AI safety. There are many examples of students who are something like this: they know what alignment faking is, read LessWrong, know who Neel Nanda is, know what METR is, have done an interp project, etc. But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.

This may be because they don’t really care much about AI safety and they just like to hang out with EA/rationalist types. An alternative reason (which I think is more likely) is because AI safety basics are something you learn and don’t exercise. When you do a BlueDot reading group you hopefully learn AI safety basics. But when doing interp experiments or your first SPAR research project, you think about specific empirical questions, not the orthogonality thesis. You don’t think about the basics and you don’t internalize them and certainly cannot reason about them.

Instrumental convergence, inner alignment, reward misspecification, etc. are our “trees are made out of air” or “the Roman Empire was 27 BC to 476 AD”. But lots of people in AI safety programs or those applying for them don’t know the basic facts. They know a lot of specific things but somehow not the foundational things.

This strikes me as a genuine failure mode. A lot of focus goes into having great fellowship programs and university groups but some conceptual knowledge seems to be slipping through the cracks. I hope the next generation of AI safety researchers have all of the conceptual knowledge of earlier researchers and more.

I’ll leave you with an additional confession. 6ish years ago, before I started college, this was probably me, at least partly. I understood the basic alignment problem but understood it mostly through outer-alignment issues and didn’t fully internalize the difficulty of the problem until I started college. That was roughly 4 years ago when I moved into the UChicago dorms. From the very beginning, I was a CS major because I wanted to be an AI safety researcher. A few hours ago I turned in my last paper, and now I’m done.

My primary reflection is this: I would not have become a CS major, would not have worked so hard, would not have been so laser-focused on AI safety if I didn’t actually understand it. I would have got distracted and ended up probably on Wall Street or worse. This is because knowing a problem tells you why you should care. So if there is one reason to embrace the basics, it is that. There is so much fucking power in actually understanding something.