This is a timed post. Every 5 minutes while writing this post, I had to stop to do 13 push-ups, and when I could no longer complete my required number, I had to upload it. My friends thought this would be a fun challenge, but it means that it will likely be less polished than some of my other output.
Summary:
Many people moving into AI safety are told that to succeed, they should read a lot and “try to develop takes.” However, it’s important not to forget why knowledge is important: it helps you do better things in the real world.
Epistemic status: Pretty confident in the main claim (you should focus on learning the most valuable things), but less confident in the specifics of what things are the most valuable. I apologize if my language is too strong at times.
“I dunno; it seems good to know things?”
The world is filled with ambitious young people, some of whom care about AI risks. Throughout my summer, I’ve gotten to talk to many of them. They visit San Francisco and Berkeley to talk to all the cool AI safety workers, or they go on surprise trips to work on high-impact, time-limited opportunities. Throughout this process, many of them look for jobs, and if they can’t find them, they seek out advice on how to get good enough to land one.
A common piece of advice for how to get a job in AI safety is “read a lot and try to develop good takes.” This advice is featured very broadly, even for people who never intend to take a technical research role:
Matt Beard: "You don't have to be confident, but you should have something to say. It's fine to keep your views loosely held and defer to experts well, but you should take ownership over forming your own takes. Hiring managers will also want to hear your thoughts and reasoning."
Kairos: “Succeeding in (generalist) roles requires real mission alignment and enough context to spot high-EV opportunities that someone without that background would miss.
As this knowledge propagates, I worry that many people who hear this advice lose track of why it’s important. Many people sharing this advice don’t spend a lot of time explaining why it’s valuable for everyone in AI safety to have their own takes.
It’s obvious why you would want to know things if you’re trying to become a knowledge expert: knowledge experts are people who know things. However, if you’re trying to become a competent chief of staff, event manager, or field builder, or you want to take one of the many other neglected generalist or org scaling roles, it’s not as obvious why you should read everything. It’s good to have an explanation for why what you’re doing is impactful, but when I ask newcomers why they think it’s important to read, I often get answers like, “I dunno, it seems good to know things?”
I worry about this. When people don’t have a good picture of why they’re doing something, they might overestimate something’s impact and miss its end goal. If you want to do good things, this is important to avoid.
Knowledge is only as valuable as what it allows you to do.
Knowledge is valuable because it helps you make better decisions. Orgs want people to be knowledgeable because people who know specific things about AI safety can do impactful things, not because storing more facts about AI is intrinsically valuable. You should focus on gaining the knowledge and tools that help you make better decisions.
To do this, I recommend using a process I call directed reading: identifying the most important questions that will help you make better decisions and doing everything you can to get answers to those. This doesn’t necessarily have to be reading: it can be listening to a podcast, talking to a friend, or chatting with an LLM. But the primary goal is to resolve your most important uncertainties.
Here are some examples of uncertainties you may have:
If you’re still not sure if AI safety is the most important thing to work on, you can ask yourself, “What is the most important problem in the world to work on?” I don’t think people ask themselves this question enough.
If you’re new to AI safety and interested in learning all the important “context” that employers keep asking for, you can ask yourself, “Why is context valuable, and how can I most quickly answer the set of important questions that having context answers?”
If you want to start some impactful projects to build your skills, you can ask yourself, “What projects give me the most valuable skills, and what are the biggest needs in AI safety right now?”
If you want to figure out what the biggest needs in AI safety are, you can ask yourself, “What does winning in AI safety look like, and what do we still need to do to make it to the finish line? In other words, what’s my theory of victory?”
If you hear that we need more generalists in AI safety, and you’re interested in solving this problem, you can ask yourself, “What problems are causing people to believe that we need more generalists, and how can I best address those specific problems?”[1]
If you plan to pursue any project, you can ask yourself, “Is this project really on the critical path to making AI go well? Do I have a coherent theory of change? Am I addressing a real problem?”
If you don’t want to get forever bogged down by infinite questions, you can ask yourself, “When is it safe to defer?”
To answer many of your questions, you’ll probably need to find answers to smaller and more fundamental questions. On its surface, your process might seem to replicate the original “read a lot of things” advice. However, by focusing on what you’re actually hoping to achieve with your knowledge search, you can develop critical skills much faster than you would by passively consuming the next 80,000 Hours podcast episode.[2]
You can also double-check your learning processes. If the knowledge you’re getting doesn’t help you make better decisions, you should be willing to quickly pivot to trying other things. Learning things does not intrinsically save lives. Misalignment risk isn’t reduced by the number of books you’ve read from cover to cover. Don’t waste time memorizing social passwords and storing fake knowledge.[3] Do what will help you do good things in the real world.
Other things to keep in mind
While focused learning is important, I don’t think directed reading is the best way to solve all of your problems.
First, while directed reading can be one of the best ways to resolve your largest uncertainties, you don’t need to spend all of your time skimming posts to find only the answers you need. Reading things you enjoy is pretty awesome, and you shouldn’t force yourself to spend all your time engaging with material that feels like homework. Find ways to integrate your study time into your routine, but don’t forget to do the things that make you happy.
Second, pure information consumption might not be the best way to gain skills and understanding. Writing can be a valuable tool to help you internalize and understand various ideas, as well as keep track of any uncertainties you might still have.[4] Additionally, if nobody’s written about the thing you want to say, you should consider polishing your notes and publishing them online. If it helps, you can assemble a group of friends and do a timed writing challenge to publish the scraps you have lying around.
Finally, you might be completely wrong about what you think you should be studying, or you might be missing some valuable new leads by continuously taking a focused approach to learning. To address this, it can be valuable to introduce some randomness into your media diet: find some time to read things (especially highly recommended things) that you don’t expect to be helpful. However, I suspect that most people aren’t spending enough time trying to learn things directly.
In the immortal words of the great Thomas Henry Huxley (aka “Darwin’s Bulldog”), “The great end of life is not knowledge, but action.”
Time stamps:
By the end of the first draft, I had done 390 push-ups.
I think I wasted about a week because I didn’t ask myself this question. These kinds of “minimal trust investigations” are valuable for becoming highly talented in a particular domain.
No hate to 80,000 intended; I think these podcasts are great informational resources. However, if you have the opportunity to sit and read, I think you should skim the transcript and find the parts that answer your questions.
This might make it easier to get a job, but if you try to study these things specifically, you’re missing the thing that actually helps you become impactful
I'll agree that learning is a means to action/decision-making. And I agree that ideally, one would focus their learning on the most important problems, etc. But I have three concerns here:
I've found that focusing on "the most important problem" is not a great foothold for research, especially because it neglects personal fit. And without wrestling with the technical side of these problems yourself to develop a deep intuition of why a problem is most important, the default state is deferring. And having a novel take without full technical understanding is usually going to be bad for a new entrant. Even if it's not a bad take, a new take could signal misalignment or misunderstanding to someone in an org who doesn't have time to evaluate it on its merits.
If this advice is for generalists wanting to support AI safety, however, I think it misses the meta layer. An org needs to be able to tell from afar that you have the right context/understanding of the problem, capability, alignment, and that you--as a generalist who is entering from outside of the effort--will not immediately abandon the effort for something new and shiny. Even if you develop a deep understanding, you need to be able to show it, and show it in a trustworthy way.
I think the most valuable knowledge for generalists to have would be tacit knowledge of how the AI safety community and orgs actually work/operate in the real world, because that's the scaffolding on which you can identify areas for improvement and high value actions. Even though orgs make an unusually strong effort to put out relevant information online, it's still not enough about actual dynamics. Getting hands-on experience with work in an org seems like it would help resolve both 1 and 2 above. But of course, this is a chicken and egg problem.
This is a timed post. Every 5 minutes while writing this post, I had to stop to do 13 push-ups, and when I could no longer complete my required number, I had to upload it. My friends thought this would be a fun challenge, but it means that it will likely be less polished than some of my other output.
Summary:
Many people moving into AI safety are told that to succeed, they should read a lot and “try to develop takes.” However, it’s important not to forget why knowledge is important: it helps you do better things in the real world.
Epistemic status: Pretty confident in the main claim (you should focus on learning the most valuable things), but less confident in the specifics of what things are the most valuable. I apologize if my language is too strong at times.
“I dunno; it seems good to know things?”
The world is filled with ambitious young people, some of whom care about AI risks. Throughout my summer, I’ve gotten to talk to many of them. They visit San Francisco and Berkeley to talk to all the cool AI safety workers, or they go on surprise trips to work on high-impact, time-limited opportunities. Throughout this process, many of them look for jobs, and if they can’t find them, they seek out advice on how to get good enough to land one.
A common piece of advice for how to get a job in AI safety is “read a lot and try to develop good takes.” This advice is featured very broadly, even for people who never intend to take a technical research role:
As this knowledge propagates, I worry that many people who hear this advice lose track of why it’s important. Many people sharing this advice don’t spend a lot of time explaining why it’s valuable for everyone in AI safety to have their own takes.
It’s obvious why you would want to know things if you’re trying to become a knowledge expert: knowledge experts are people who know things. However, if you’re trying to become a competent chief of staff, event manager, or field builder, or you want to take one of the many other neglected generalist or org scaling roles, it’s not as obvious why you should read everything. It’s good to have an explanation for why what you’re doing is impactful, but when I ask newcomers why they think it’s important to read, I often get answers like, “I dunno, it seems good to know things?”
I worry about this. When people don’t have a good picture of why they’re doing something, they might overestimate something’s impact and miss its end goal. If you want to do good things, this is important to avoid.
Knowledge is only as valuable as what it allows you to do.
Knowledge is valuable because it helps you make better decisions. Orgs want people to be knowledgeable because people who know specific things about AI safety can do impactful things, not because storing more facts about AI is intrinsically valuable. You should focus on gaining the knowledge and tools that help you make better decisions.
To do this, I recommend using a process I call directed reading: identifying the most important questions that will help you make better decisions and doing everything you can to get answers to those. This doesn’t necessarily have to be reading: it can be listening to a podcast, talking to a friend, or chatting with an LLM. But the primary goal is to resolve your most important uncertainties.
Here are some examples of uncertainties you may have:
To answer many of your questions, you’ll probably need to find answers to smaller and more fundamental questions. On its surface, your process might seem to replicate the original “read a lot of things” advice. However, by focusing on what you’re actually hoping to achieve with your knowledge search, you can develop critical skills much faster than you would by passively consuming the next 80,000 Hours podcast episode.[2]
You can also double-check your learning processes. If the knowledge you’re getting doesn’t help you make better decisions, you should be willing to quickly pivot to trying other things. Learning things does not intrinsically save lives. Misalignment risk isn’t reduced by the number of books you’ve read from cover to cover. Don’t waste time memorizing social passwords and storing fake knowledge.[3] Do what will help you do good things in the real world.
Other things to keep in mind
While focused learning is important, I don’t think directed reading is the best way to solve all of your problems.
First, while directed reading can be one of the best ways to resolve your largest uncertainties, you don’t need to spend all of your time skimming posts to find only the answers you need. Reading things you enjoy is pretty awesome, and you shouldn’t force yourself to spend all your time engaging with material that feels like homework. Find ways to integrate your study time into your routine, but don’t forget to do the things that make you happy.
Second, pure information consumption might not be the best way to gain skills and understanding. Writing can be a valuable tool to help you internalize and understand various ideas, as well as keep track of any uncertainties you might still have.[4] Additionally, if nobody’s written about the thing you want to say, you should consider polishing your notes and publishing them online. If it helps, you can assemble a group of friends and do a timed writing challenge to publish the scraps you have lying around.
Finally, you might be completely wrong about what you think you should be studying, or you might be missing some valuable new leads by continuously taking a focused approach to learning. To address this, it can be valuable to introduce some randomness into your media diet: find some time to read things (especially highly recommended things) that you don’t expect to be helpful. However, I suspect that most people aren’t spending enough time trying to learn things directly.
In the immortal words of the great Thomas Henry Huxley (aka “Darwin’s Bulldog”), “The great end of life is not knowledge, but action.”
Time stamps:
I think I wasted about a week because I didn’t ask myself this question. These kinds of “minimal trust investigations” are valuable for becoming highly talented in a particular domain.
No hate to 80,000 intended; I think these podcasts are great informational resources. However, if you have the opportunity to sit and read, I think you should skim the transcript and find the parts that answer your questions.
This might make it easier to get a job, but if you try to study these things specifically, you’re missing the thing that actually helps you become impactful
I also like Andy Masley’s blog post about this