Instrumental convergence, inner alignment, reward misspecification, etc. are our “trees are made out of air”...
fyi, I just completed the AGI Strategy Course from BlueDot and none of instrumental convergence, reward specification, the orthogonality thesis, or wireheading were taught there. Thus, when a few of the intro scenarios involved misaligned AGI (instead of bad actors) doing nefarious things, the students were very surprised. One student wondered out loud, "Why would AI even hack into all our infrastructure? It makes no sense," and our facilitator pivoted to talking about terrorist groups.
These core concepts (explained to me through Rob Miles AI Safety videos) are what got me interested in AI Safety. For me, they are on par with the more Overton-friendly bad actor risks.
As an aside, I have been confused why AI experts on talk shows often decline to explain the core concepts that make alignment so tricky. I think there is an easily accessible version of the instrumental convergence conversation that should be making the rounds.
9An alternative to Bluedot's course is the https://ai-safety-atlas.com/ , which does present the case for risks with the standard arguments made on lesswrong, and which I believe is the sota end to end written explanation of catastrophic risks.
Note: I'm one of the authors
Raising situational awareness is probably one of the most effective ways to make progress. It’s important to lay out object-level arguments clearly, and to support them as much as possible both theoretically and empirically.
Trying to make moves that ostensibly address A while the actual threat model in mind is B is epistemically bad, and from my experience, not even more effective in practice. It does take a bit of courage, but in practice the idea that people won’t take you seriously is mostly false, it’s largely a mental barrier. If you’ve genuinely understood the problem at the object level, and you make a real effort to transmit that understanding trying to maximize information transfer and helping others form the right intuitions, then it doesn’t come across as weird at all (even the contrary). Be honest, serious, and rigorous, don’t play a political game, people do understand.
Fyi, i stopped presenting the theoretical concepts myself to policymakers, because we have enough incidents/concrete experiments that make the case without needing the theoretical scaffolding.
Lack of conceptual understanding of the basics seems to me like a major reason why people keep on doing activities that sound like "AI Safety" but are likely making things worse.
It's also what we're trying to change with Lens Academy.
Interesting hypothesis that such basics might be less understood because they're inherently less trained in practice while doing research. Seems plausible.
I do also think the AI Safety education space has a share in this, focusing too much on prosaic empirical methods at the cost of strategy and conceptual fundamentals.
Shout outs to AFFINE and Iliad Intensive for also emphasising the basics (as far as I can see)
3
1people keep on doing activities that sound like "AI Safety" but are likely making things worse.
What kind of activities are you thinking of specifically?
Hmm, sth like doing ML research or creating tooling that's dual use without really attending to the idea that it might be dual use and without a clear theory of change. Sorry I don't have something more specific for you here.
Taking this literally, trees being made out of air implies a lot of useful facts about the world:
- There's carbon dioxide in the atmosphere
- Trees (and therefore most Earth-like macroscale herbivores, and therefore basically most macroscale Earth-like life) probably require atmospheres, with at least moderate concentrations of CO2
- So there is no life on the Moon, Mercury, and the gas giants (nor most of the moons of any of the gas giants, at least of terrestrial nature). There is probably also no life on Mars (atmosphere too thin), or (with some data about the extremely high atmospheric pressure on its surface) Venus, at least right now.
- Since plants are the lowest part of the food chain, this also implies most macroscale animals are made out of carbon
- Since trees do not appear to be made out of mostly oxygen (especially once you determine the chemical formula for cellulose), you can also gather that trees eject some amount of oxygen from CO2 back into the atmosphere (else you would be violating conservation of mass)...
- which implies plants (or at least trees, though it would be easy to figure out that other plants are also made out of mostly cellulose) produce lots of, or maybe even most oxygen on Earth
- This also implies that there was a time, before trees or photosynthesis, where Earth's atmosphere wasn't oxygenated, and was mostly CO2, potentially from volcanism
- If your civilization is mostly powered on the combustion of dead plant-matter (whereas such combustion is responsible for most economic activity), and if you have knowledge that CO2 traps heat, you can conclude that it is likely that the planet will heat up, conditional on civilization's continued progression
- What if we plant many trees, in order to reduce the amount of CO2 in the atmosphere? What if we build a better tree, since natural selection definitely didn't select for maximal CO2 usage - it selected for IGF, and intentionally inefficiently using CO2 would go against IGF, and obviously all easy ways for trees to efficiently use CO2 would have already been selected for via IGF, because trees are made out of air (does similar also apply to human intelligence, as well?)
- Trees (and therefore most Earth-like macroscale herbivores, and therefore basically most macroscale Earth-like life) probably require atmospheres, with at least moderate concentrations of CO2
- Nature can construct mesoscale artifacts of the volume and mass of redwood trees and blue whales, and have said artifacts[1] cover most of the planet; with only naive collection of CO2 on the former case; or, in the latter case, indirect naive collection of CO2, coupled with rudimentary intellect
- Yet humans can't (or at least not yet). In addition, the mechanisms used to produce wood seem unintelligently designed (along with many other such biological mechanisms). Why?
- If it is possible for our intellect to be 'designed' by such a dumb (and simulatable!) process as natural selection, could it be possible to design a thinking machine, potentially by simulating a highly compute intensive process selecting and rewarding the best 'programs' internally (though such programs need to be fundamentally continuous, else the reward function would be too sparse) via increasingly massive clusters of number-crunching machines?
- Obviously this will almost certainly result in an naively uninterpretable mess of heuristics; how could that be fixed?
- What about overfitting to the continuous proxy, and not the true metric we care about - like morality, or not turning the world into computronium?
- If it is possible for our intellect to be 'designed' by such a dumb (and simulatable!) process as natural selection, could it be possible to design a thinking machine, potentially by simulating a highly compute intensive process selecting and rewarding the best 'programs' internally (though such programs need to be fundamentally continuous, else the reward function would be too sparse) via increasingly massive clusters of number-crunching machines?
- With sufficiently advanced technology, you could construct macroscale objects (with masses of at least 4 million pounds, and volumes of at least 80,000 cubic feet) out of entirely atmospheric carbon, and such objects can even self-replicate to cover much of the world's surface[2]
- How? How nature accomplishes this feat seems intimately connected with much of nature's doings being done in the nanoscale. After millions (or billions) of inferential steps, you would likely arrive at something like Drexler's molecular nanotechnology, with its associated post-scarcity (at least on the human-scale) economics, directly attributable from the fact that nanoscale systems (due to their smallness) can self-replicate on primitive feedstocks on the order of hours or minutes, not months or years
- Yet humans can't (or at least not yet). In addition, the mechanisms used to produce wood seem unintelligently designed (along with many other such biological mechanisms). Why?
Obviously much of this speculation of how 'trees are made out of air' allows you to predict that Drexlerian assemblers, constructed via a misaligned superintelligence, will convert the Earth into computronium is benefited by a lot of hindsight; if you told a 15th-century peasant that 'trees are made out of what Plato called aer', they would almost certainly say 'neat fact' and not suddenly refactor their entire epistemology to be based on that factoid, even though I think they should.
But knowing such facts with as much general utility as 'trees are made of air' are very useful to gain a highly sample-efficient epistemology; and you very much want sample efficiency when trying to align AI, because aligning sufficiently advanced AI is a textbook example of a problem with extremely sparse rewards (this follows naturally from the fact that the entirety of humanity may only get only one shot to align said AIs).
2
1
1I have a fig tree in my front garden. It lives in a comically small pot of earth (relative to its size). Every winter I cut it back, and every spring it explodes in size. Without fail, every year, at least one person stops to ask how the tree gets enough from the pot to grow so much, and every year I explain that trees mostly eat air, not soil. These people are often keen gardeners! But it's not a commonly known fact.
2Very similar to the idea that in a car (with an internal combustion engine) you only carry about 15% of the car's fuel in the fuel tank... the rest is the air.
One fun corollary that is unintuitive to most people: when you lose weight, how does it exit the body? Answer: you breathe it out.
Even if you don't lose weight, much of what you eat you exhale.
In some sense, our bodies are processing facilities for tree food, and before the tree consumes it, it is temporarily stored in the warehouse for tree food: the atmosphere!
The point I make here is that having a deep understanding of AI safety helped keep me focused in my undergrad. That is, deeply understanding what the problem is and why it is difficult keeps one convinced that they should continue working on the right kinds of things.
Also though: A lot of pretty profound ideas come from pretty simple intuitions about foundational concepts. E.g., my understanding is Alan Turing came up with the idea of a Turing machine by reasoning about how a human computer would calculate something. But then the Turning machine model is a really powerful mathematical tool for all sorts of things. Similar things (where a simple intuition motivated a key discovery) happened with neural networks and possibly special relativity if my understanding of the history is correct. To me, understanding the basics is a prerequisite (or maybe the main prerequisite) for all of this.
thanks!
these examples are much more convincing than the essay's. i haven't looked at each closely, but if these several inventions are all due to deep understanding, that is indeed a recommendation for deep understanding.
were i not already convinced of the importance of deep understanding in advancing the frontier of human knowledge, this comment would be a good starting point.
may i ask: why not use these examples in the original essay? the essay touts results like "know dates", "save face if documentarians come round", "finish a degree / turn down quant offers". these are -- lightly now -- relatively less interesting to me than, like, "invent turing machines".
if i model the goal of the essay's author as "try to convince me that deep understanding is worthwhile", then i am very confused. why would such an author appeal to the examples in the essay, if they could have reached for the ones in the comment?
do i misunderstand the essay's goal?
I see the goal here is to give an intuitive feeling for how basic gaps in knowledge can emerge. I see this happen all the time in AI safety and as someone who runs an AI safety program, I actually find this example more interesting than the Turing machine example. But that's a personal thing and although I think this kind of lesson should hold generally, convincing the reader of this isn't the core thing I'm interested in.
Understanding the basics of the carbon cycle is pretty critical - misunderstanding of which, e.g., drives much of the idiocy around climate chang being unsolvable.
The power to correctly reason about trees in a wider range of circumstances, which generalizes to all organisms that do photosynthesis and their effect on their environment? Also a generalization that the physical properties of a thing don't always intuitively match the properties of "where it came from" - trees are solid and brown like ground, and so it will seem like there's probably a strong relation between the content of a tree and the content of the ground, And the roots really look like they're sucking important things from the ground, but guessing that trees came mostly from ground would be wrong, and this "the world is weird and your intuitions can be wrong" lesson generalizes, and could spark thoughts of what other cool things chemistry might be capable of, as well as making one more careful about making similar guesses in the future.
1Random anecdote: A few months ago I had an argument with a PhD student (and blogger and EA forum participant) whose mental model was something like: every second that a plant is alive, it is converting CO2 to O2, and every second that an animal is alive, it is converting O2 to CO2. Therefore, if there are more animals than plants on Earth for a long enough period of time, those two processes will be out of balance, and more and more O2 will get converted on net to more and more CO2, and eventually we won’t have any oxygen to breathe. (Which is absurd, and he probably would have understood that if he had understood that “trees is air”.)
(To his credit, he noticed that nobody else was worried about running out of oxygen and therefore he must be somehow confused, and was trying to figure it out.)
Actually can you explain this more? In the limit where there are no photosynthesizing organisms and animals somehow didn't die from lack of food first wouldn't we eventually run out of oxygen?
You can only convert an O2 to a CO2 by adding a carbon atom. The key question is: where did that carbon atom come from? That’s the part where “trees is air” is important.
One possible answer is: the carbon atom could come from the biosphere (trees, and other plant matter, plus animal bodies etc.). But there’s orders of magnitude fewer carbon atoms in Earth’s entire biosphere today than there are O2 molecules in the atmosphere. So you could combust the entire biosphere of Earth, to ash, all at once tomorrow, and that would sure be bad for climate change but it would be a negligible change in atmospheric O2. If instead of combusting the entire biosphere tomorrow, animals instead eat down the whole biosphere over many years, until there’s no plants left, then the animals themselves starve to death, and the scavengers eat the corpses, etc., then that’s the same number of carbon atoms getting attached to O2’s, i.e. all the carbon atoms previously stored in the biosphere, it’s just that they would be getting attached to O2’s more gradually.
I asked some LLM, and it guessed that if we combust the entire biosphere to CO2, that’s enough carbon atoms to reduce atmospheric O2 by <1%. If we also extract and burn all the fossil fuels on Earth, that’s still only enough carbon atoms to reduce atmospheric O2 by <10%. I didn’t check those though.
To your question, “animals somehow didn't die from lack of food first” is an incoherent hypothetical. In order for animals to get energy and create CO2, they have to be eating carbon-based food. Again, the question is: Where did those carbon atoms come from? See what I mean?
Thanks, yes I see what you mean. I hadn't thought that it would be so negligible. Thanks for explaining further.
I guess over millions of years the oxygen would deplete to practically nothing but this would be due to erosion and tectonics not animal respiration.
Very little, but realising that implies you have a gears level understanding of what's actually happening in biology rather than just being able to solve the problems you get asked in exams.
Reminds me of https://calteches.library.caltech.edu/46/2/LatinAmerica.htm
https://en.wikipedia.org/wiki/Justus_von_Liebig. Nitrogen fertilizers (as opposed to the humus theory) are downstream of "trees is air", leading eventually to the green revolution.
It's evidence of a wider array of knowledge they have not unlocked. Learning more biochemistry has put me on firmer ground when trying to evaluate nutrition facts which was an area where before I was epistemically helpless. Learning the basics takes less time than you might think and you waste less time listening to idiots (on for example nutrition). Language models are good at biochemistry, but they will imitate a dumb nutritionist rather than a nutritionist who knows biochemistry if you don't use the right vocabulary.
Curated. I appreciated this both as a general rationality lesson and the specific AI safety lesson.
I do wish it said more about how to deal with the problem of missing basics. I'm still not sure reading this what I'd get from really internalizing that trees are made of air, or, how to integrate the basics into my day-to-day in a way that would help them stick. (Or rather, I have ideas on that, but, they don't really come from the essay.).
1Some pedantry: living trees have a lot of water, so by mass the tree may even be predominantly water, depending on the tree.
Yes! This is true. I tried to say "dead tree" or "dry tree" to avoid this but reading through it again maybe it is confusing. I think I'll add a footnote to clarify. Thanks!
Wait a minute! Turning
Half of the oxygen from the
For those who are now worrying about missing some foundational aspects of knowledge on AI safety, could you point towards the first thing to read to check understanding?
1Sure!! (also feel free to DM me if you want to discuss further. This is the kind of conversation I would be very willing to have with anyone who wants to chat)
The Problem is a good intro to misalignment risk and there is some interesting discussion in the comments. [1] I find that The Superintelligent Will holds up well even if it is sort of dense and old. I have been told by people I trust that this is a good resource to start with but I haven't read it or maybe I have read it and just forgot.
I take misuse risks pretty seriously too but the argument can basically be contained in "models will be able to cause harm and people will try to use them to do that. Jailbreaks and fine-tuning attacks, etc are a thing so those bad actors may succeed in doing things like building bio weapons or something equally worrying."
The hueristic I use when undergrads ask me if they should drop out of college is: 1. can you explain instrumental convergence to me and how this fits into the extinction story and 2. can you explain the difference between inner and outer alignment (maybe with some basic help in the form of a refresher of the definitions).[2] If you can't do those things, you probably should be 100% be staying in school because you would not know what you are dropping out for. I would recommend that people reuse this test on themselves!
- ^
I think probability of extinction is less than the median MIRI researcher because I'm more optimistic about inner alignment but if the goal is to check understanding, this is a good place to start.
- ^
Maybe inner vs outer alignment isn't the right framing (I think it is but it's a source of controversy) but I still think this is a good question because it tests how well you can reason about different kinds of failure modes and how they may interact, etc.
Historical context - the human stories of science, vignettes from the climb up from animal ignorance, driven by need or curiosity - helps to make that story coherent. It was around 1620 when curiosity impelled Jan van Helmont to grow a willow tree, in a weighed pot of soil, for 5 years - then pull it up, clean the roots, and weigh both tree and soil. His conclusion was that trees are made of water. Generations later Lavoisier amended that to "air and water" - by carefully capturing and analyzing combustion products - before being beheaded in the French Revolution (1794).
Jan Baptist van Helmont was not beheaded. You are likely thinking of the famous French chemist Antoine Lavoisier, who was executed by guillotine during the French Revolution in 1794. Helmont died in his bed from a lung infection in 1644.
I believe the tree experiment was proposed by Francis Bacon, though I don't recall if he was before Helmont. It seems to have been one of the first experiments of the renaissance, coupled with an ideology that focused on experimentation.
One student, upon being told that most of the mass in the wood comes from and not the dirt in the ground, said “that's very disturbing and I wonder how that could happen.” And that’s an MIT graduate.
Yes, and some Harvard grads thought the seasons were caused by varying distance between the Earth and the Sun. However, this source was 1987, and judging by the VHS cassette in yours, it was probably also old. So I suspect that there would greater awareness now with climate change that trees soak up CO2, and that both unis have made more of an effort to have their students know things like this because of the bad press. Also, I expect that there was some cherry-picking. However, I agree with your overall point that many well educated people don't know basic facts about the world.
2I help run the Pivotal fellowship. We had a discussion activity where we took various statements and agreed/disagreed ("change my view" activity). Around 2/3 of fellows were pessimistic about current methods being on track to align or control superintelligence.
Would this update you a bit against your original point in the post? I'm not that sure very detailed understanding of Instrumental Convergance is needed for getting a decent intuition for why ASI alignment is going to be hard
Curious to hear your takes and if you have ideas for how to elicit the fellows opinions/get them to engage with the ideas in a useful way.
To make sure I understand correctly: the fellows don't know what instrumental convergence is but they think alignment is hard (probably implied by not feeling like we are on track to solving it)?
I havent asked them whether they know about Instrumental convergence, but i'd guess at least 50% of them dont - ppl that joined the field 1yr ago or less and mostly did SPAR/Bluedot generally wont be very MIRI pilled.
Nicely written. Most of us are trained to be specialists. Being a generalist is, in most cases, rewarded less than being a specialist.
I do know about Roman History and can answer those questions you can't anymore. In my opinion most of the weight in wood comes from solar energy, not from air. I can answer questions about psychosis, biology, popular culture, quantum mechanics, history and even know what breathwork practicioners are (revelant!). And I can also answer questions about AI safety, but I don't know who Neel Nanda is. I'm a generalist pur sang.
Most people learn things primarily out of neccesity or out of passion, and, in many cases, a solid foundation is not a neccessity to be an effective specialist. The issue is more that we turn to specialists when it comes to questions that concern foundational issues, somehow assuming that being a great specialist also implies being a foundational generalist too. But reality is that most of us are not Da Vinci's.
in my experience few people put in effort to make a model of the world beyond what they need, precisely because they don't need to. If you solve the lack of neccessity, you can solve the lack of foundations.
When you do a BlueDot reading group you hopefully learn AI safety basics.
How sure are you about this?
[Epistemic status: Rumor
Someone told me that the BlueDot team don't believe in full AI X-risk, but instead teach smaller risk, e.g. misuse risks. I have not verified this, or looked at their curriculum to see what they teach.]
But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.
I'm not surprised about this. But I'm leaning towards that the problem is that they never really leaned it, rather than that they learned it and forgot.
How sure are you about this?
I say "hopefully" for a reason. I am not under the illusion that everyone who completes any intro curriculum will learn AI safety basics. With that being said, I just skimmed the BlueDot technical AI safety curriculum and it seems fine. I think it was better when I did it a few years ago but its clear that they have optimized to make it more approachable even if it is less comprehensive and maybe this was the right call -- I'm not sure.
Someone told me that the BlueDot team don't believe in full AI X-risk
I would be pretty surprised if this was true, based on two people I know that have worked at BlueDot (both people who are very impressive + mission-aligned and I would hire if I could). I would want to know more details and in the meantime I don't want to speculate because my prior is that the BlueDot staff do not feel this way and I generally trust BlueDot to make reasonable decisions.
But I'm leaning towards that the problem is that they never really leaned it, rather than that they learned it and forgot.
I hope you are wrong but unfortunately I think you could be right in some/many/most cases. I think what I describe in the original post still happens but the problem may be a lot larger (in which case we may want to rethink how we do intro fellowships).
I am also curious what you think a good solution looks like! If you have ideas I may try to make it happen.
1I'm not sure what the solution is.
I'm not surprised that you've found that lot's of people don't know the basics, but thinking about why I'm not surprised, it's because I've heard this before several times. But I am confused. The basics is really not that hard. "If we build something smarter than us, the default outcome is that that thing, not us, is then in control". There are also a few arguments why alignment is hard, but not that much.
Maybe part of the solution is for everyone to taka a day (or more) and just try to solve alignment? To really notice the difficulties themselves?
Possible the problem with alignment basics and the tree thing, is that the people asked are trying to remember the answer rather than derive the answer. Possibly because they think the answer is difficult.
I know that wood is mostly made out of air, specifically mostly made out of CO2 from the air. I know this because I know that (other than water) life is mostly made of carbon, and I know that trees get's their carbon from the air, because that's what photosynthesis is. It's easy if you remember to think. (Additionally you can notice that large trees don't create equally large holes in the ground, so it can't mainly be soil they eat.) Someone who get's this wrong probably don't know less biochemistry than me, but are doing something else wrong.
I also read the Aeneid while studying Latin. It isn't hard to recognize the symmetrical structure of the line 'spem vultu simulat, premit altum corde dolorem' in Book 1, line 209. You might also know—whether from being taught or simply by intuition—that this reflects Aeneas's internal state at the time. To be fair, in the context of studying Latin, that qualifies as 'basic knowledge.' I don't want to deny that this played a huge role in helping me read the Aeneid to the very end. That specific knowledge wasn't actually all that useful for getting a high score on the AP Latin exam. Ultimately, however, that seemingly trivial piece of information helped me finish the book, and it undoubtedly gave me the motivation to master the rest of the material needed to get a good grade. I'm not entirely sure, though, if my experience relates to the aspiring AI alignment researchers you mentioned.
1Its a beautiful line! Not sure if it relates to this essay but I find it highly relevant to our current moment.
Thank you for saying that. In fact, I believe this is exactly why many colleges still utilize a 'holistic review' process in their admissions, evaluating essays and extracurricular activities rather than just relying on numbers.
Numbers like GPAs and AP scores certainly demonstrate a student's academic capabilities. However, through essays, colleges want to see how foundational knowledge—such as the fascinating yet scientifically obvious truth that a tree is mostly made of air—supports and shapes a person's life values. College is a marathon, not a sprint. And I believe it is precisely this kind of wisdom that helps us complete that marathon.
I am currently 19 years old and will be starting college as a freshman this September, which is probably why your writing resonated with me so deeply.
I recently facilitated the AISF course again (technical, governance and strategy courses). If you've been in AIS for a while and you want to revisit the basics, there's certainly worse ways of doing it.
At the risk of adding oil to the fire, I have witnessed conversations that went like this:
"Should we add a slide on instrumental convergence and the orthogonality thesis ?"
"No. It's outdated and too abstract, and it will nerd-snipe people into the old MIRI agenda and agent foundations, which was a waste of time and is completely useless. Talk about empirical stuff [and focus on governance]"
"But don't you think it helps people think more clearly about things?"
"Not sure. I think it confuses people more than helping them"
It seems clear to me that some people very adamantly reject the "Rob Miles stack" and believe it is wrong (and harmful?), and that those people (the less famous ones, at least) are not remarkably vocal on LessWrong (and tend to dislike posts like this one, as well as LessWrong in general). Those people include both people with significant X-risk concerns and people for whom X-risk is not a significant concern.
I now transparently explain this disagreement (on the scientific level) when presenting the alignment problem to new audiences. The attendance usually finds it helpful and enlightening, as researchers often (accidentally, hopefully) mask the existence of competing viewpoints during introductions to the topic, which makes people somewhat confused depending on how they were introduced to it so far.
Tangentially, here's a little Feynman interview excerpt in which he talks about trees coming out of the air, and how they store sunlight in a sense.
I think it goes both ways. As you said, knowing "trees comes from air" may just be an irrelevant fact for solving hard problems. I think that the specific questions you make in the post are similar, not in that they are irrelevant facts, but they are motivational questions that may not hinder one's ability to solve hard challenges in AI safety. Of course, it might be better to have a solid motivation for AI safety research, as it can better help you solve the problems.
I will correlate personal motivation to the irrelevant subject-specific facts, in that, in exceptional cases, these facts may prove to be useful. Similarly, in some cases, solid personal motivation may put you in a stronger position than those without.
Education, except maybe at post graduate level, is entirely about learning how to do stuff or learning about things that may help you to do stuff. AI safety isn't like that. You cannot learn AI safety in the the sense that is usually discussed here because nobody knows how to do it. All you can be taught is a bunch of stuff that is known not to work. It is taught because it will help get you a job, and is good enough for that purpose and that purpose only, not because it will allow you to create an aligned AGI.
Realising that trees are mostly air requires the knowledge of multiple scientific facts and methodologies; carbon and oxygen are heavier than hydrogen, measuring carbon input/output, usage of isotopes to make sure we are registering the same carbon in our input/output experiment etc.
Van helmont's tree weighing experiment (1640s) was, in my opinion, more complex than the fact that 'trees are made of air'.
Thus, a paradox is identified.
Maybe I'm getting caught up in your example. Though i understand the substance of your thought, "experts are engaged in advanced methods yet lack foundational insights."
The result of the tree weighing experiment, the realisation that tree comes from air, is an output.
So, the foundational knowledge, atleast in this example, is a derived fact. A fact that requires more knowledge to derive it than what it provides us.
Thus, to understand a foundation deeply, one must understand the complex mechanisms that built it.
If researchers are not aware of the basics, it could be because the underlying variables which govern those basics are still being modelled or contested...
So, when researchers are asked "Why AI is an existential risk" it's not necessary that they lack foundational insight. It could be because the "basics" require a complex chain of theoretical/empirical reasoning (e.g., capability scaling, mesa optimisation, failure of containment) to derive it, and may genuinely cause researchers to struggle to provide a coherent answer.
"Orthogonality Thesis", "Instrumental Convergence" etc. are theoretical frameworks and not empirical facts, unlike photosynthesis.
They are highly complex frameworks that attempt to predict complex systems that do not fully exist yet.
Conclusion
Stammering on a foundational AI safety question is not equivalent to an MIT graduate forgetting where a tree's mass comes from. The expert forgot/ignored a compressed empirical fact; the AI safety applicant is struggling to articulate a highly debated, theoretical framework that relies on a massive stack of unproven assumptions. Demanding that researchers treat these complex theoretical frameworks as simple, easily internalized axioms is risky to spirit of scientific thought.
In AI safety, this can be a serious problem.
I have had one-on-ones or interviewed dozens of students who want a career in AI safety. [...] But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.
I'd say the situation is even worse than that. I've recently had one-on-ones with researchers in AI safety governance/policy, and most of them didn't seem to have engaged with the object-level arguments. I suspect the same holds for most technical AI safety researchers, but my sample is small.
I suspect it's mainly the lack of object-level thinking about the problem that leads many orgs and researchers to a prioritization that seems miscalibrated to me. The majority focuses on misuse risks, rather than existential risks from loss-of-control scenarios, even though the expected impact is far greater.
Trees are air actually came from people reasoning across fields (physicians, ministers, chemists, not botanists). I buy the missing-basics point, but what's the AI safety equivalent?
I think the cross-domain thing is confounded by basically all science pre modern era being done by polymaths. As an example I do not think you can validly reason that searching for bible codes are important for inventing calculus, aside from a general high-variance attitude to work.
But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.
Related: My hobby: running deranged surveys
dry wood is mostly CO2
FWIW, you appear to be confusing two claims:
"Most mass in trees was previously part of the surrounding air"
"dry wood is mostly CO2"
From a US Forest Service website:
Wood is best defined as a three-dimensional biopolymer composite composed of an interconnected network of cellulose, hemicelluloses and lignin with minor amounts of extractives, and inorganics. The major chemical component of a living tree is water, but on a dry weight basis, all wood cell walls consist mainly of sugar-based polymers (carbohydrates, 65-75%) that are combined with lignin (18-35%).
https://research.fs.usda.gov/treesearch/42245
"Trees are mostly made of air" (direct quote from post title) is more clickbait headline than scientific fact.
1
At the risk of embarrassing myself, I’ll share a confession.
For context, I took five years of Latin: four in high school and one in college. In addition to learning the language, all my Latin classes taught a lot about Roman history. Emperors, internal politics, Caesar, etc. I was always learning some random bag of facts about Roman history. In high school, I won the award for top Latin student in my graduating class. So I wasn’t a bad Latin student.
Here’s the confession: I somehow don’t even vaguely remember the rough timespan the Roman Empire existed. Maybe Jesus time? I know he was killed by the Romans (is that right?). Were they around for a long time after? A long time before that? When was Romulus and Remus allegedly fighting? Virgil wrote the Aeneid when? I don’t have a clue. Despite being a kind of “Latin expert” I am missing a much more important foundational fact: when all of this was happening.
When I say trees are made out of air I’m not talking about the fact that there is a lot of empty space inside a tree (or actually anything made out of atoms). I mean something more mind-blowing.
Imagine you are holding a piece of dry wood. Where did that wood come from? A tree. Okay sure, but what are you actually holding? Where did the tree get that stuff? It turns out that almost all of the mass in dry wood comes from the
The carbon and the oxygen in the glucose (
It’s kind of unintuitive. Wood is hard and stiff and air is not even close to either of those things. But if you are a biologist or know some basic chemistry you would think that this is the kind of obvious thing you would want to know. At the very least you would expect the best math and science students in the world to be familiar with such basic biology. But take a look at this video of a documentary film crew asking MIT graduates where they think wood comes from:
One student, upon being told that most of the mass in the wood comes from
“Trees come from mostly air” is pretty fundamental for biology because it follows from photosynthesis which is in many ways the basis for life on earth. In a sane world, every 8th grader would know that dry wood is mostly
There is this assumption that you should first learn the foundational facts of an area of study and then move to more and more specific questions and ideas. As you move up levels of classes, the foundational stuff seems more and more basic and less and less relevant. Some stuff is continually hammered in because it’s useful background knowledge. In a perfect world, this helps you internalize the basics and learn how to reason about them to solve harder and harder problems.
There are some areas where, without much effort, this may work out. For example, you learn fractions in first or second grade but will probably understand them more deeply by the time you get to calculus because you need to know fractions to do calculus and all the classes that come before calculus. But not all foundational knowledge is like this! Knowing where trees come from doesn’t help you answer organic chemistry or evolutionary biology questions.
Being 1. foundational and being 2. useful for answering increasingly specific questions are different things. They are certainly not orthogonal but they are also not perfectly correlated. When 1 & 2 diverge, you get MIT grads who are confused about what wood is.
In AI safety, this can be a serious problem.
I have had one-on-ones or interviewed dozens of students who want a career in AI safety. There are many examples of students who are something like this: they know what alignment faking is, read LessWrong, know who Neel Nanda is, know what METR is, have done an interp project, etc. But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.
This may be because they don’t really care much about AI safety and they just like to hang out with EA/rationalist types. An alternative reason (which I think is more likely) is because AI safety basics are something you learn and don’t exercise. When you do a BlueDot reading group you hopefully learn AI safety basics. But when doing interp experiments or your first SPAR research project, you think about specific empirical questions, not the orthogonality thesis. You don’t think about the basics and you don’t internalize them and certainly cannot reason about them.
Instrumental convergence, inner alignment, reward misspecification, etc. are our “trees are made out of air” or “the Roman Empire was 27 BC to 476 AD”. But lots of people in AI safety programs or those applying for them don’t know the basic facts. They know a lot of specific things but somehow not the foundational things.
This strikes me as a genuine failure mode. A lot of focus goes into having great fellowship programs and university groups but some conceptual knowledge seems to be slipping through the cracks. I hope the next generation of AI safety researchers have all of the conceptual knowledge of earlier researchers and more.
I’ll leave you with an additional confession. 6ish years ago, before I started college, this was probably me, at least partly. I understood the basic alignment problem but understood it mostly through outer-alignment issues and didn’t fully internalize the difficulty of the problem until I started college. That was roughly 4 years ago when I moved into the UChicago dorms. From the very beginning, I was a CS major because I wanted to be an AI safety researcher. A few hours ago I turned in my last paper, and now I’m done.
My primary reflection is this: I would not have become a CS major, would not have worked so hard, would not have been so laser-focused on AI safety if I didn’t actually understand it. I would have got distracted and ended up probably on Wall Street or worse. This is because knowing a problem tells you why you should care. So if there is one reason to embrace the basics, it is that. There is so much fucking power in actually understanding something.
The reason I'm saying "dead tree" or "dry tree" above is that living trees can contain a lot of water mass but this analysis only focuses on the non-water parts of the tree--i.e., the hard "wood stuff."